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SUMMARY 


This  report  examines  the  benefits  of  reciprocal  agreements  for  assessing  suppliers’ 
quality  management  systems  and  considers  how  DoD  can  reduce  costs  by  eliminating 
redundant  audits  of  such  systems. 

BACKGROUND 

Supplier  oversight  processes  entail  significant  expense  for  both  industry  and  DoD. 
Best  conunercial  practice  seeks  to  reduce  this  expense  through  stringent  evaluations  of 
suppliers’  quality  system  effectiveness,  followed  by  reduced  oversight  and  long-term 
partnering.  Suppliers  that  meet  evaluation  criteria  gain  the  status  of  “preferred”  or 
“certified.” 

Currently,  companies  face  the  potential  that  their  product  and  process  quality  will 
undergo  numerous  audits  and  certifications  by  various  industry  bodies  as  well  as 
government  customers.  Audits  are  required  to  achieve  preferred  supplier  status  and 
Malcolm  Baldrige  finalist  status,  and  to  qualify  for  such  programs  as  the  Army’s 
Contractor  Performance  Certification  Program  (CP)2,  Boeing’s  Dl-9000  certification,  the 
Big  3  Automaker’s  QS-9000  certification,  and  ISO  9000  registration  and  qualification. 
But  because  there  is  no  universal  reciprocal  acceptance  among  the  various  industry 
associations  and  government  agencies,  the  savings  from  reduced  oversight  and  the 
elimination  of  redundant  audits  and  assessments  are  only  partially  realized  by  both  the 
customers  and  the  suppliers. 

IDA  was  tasked  to  determine  how  DoD  can  enter  into  reciprocal  agreements  with 
industry  quality  system  assessment  bodies  and  other  government  agencies  to  reduce 
DoD’s  own  oversight  resource  requirements. 

IDA  APPROACH 

IDA  first  defined  the  numerous  terms  used  in  this  area  of  study.  The  terms  are  not 
consistently  used,  and  this  was  a  problem  we  had  to  overcome  throughout  the  study.  We 
described  and  compared  several  quality  system  assessment  programs  and  quality  award 
programs  to  see  their  differences  and  similarities.  We  used  ISO  9000  or  its  U.S. 
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equivalent,  ANSI/ASQ  Q9000,  as  the  baseline  for  quality  system  assessment  programs 
and  the  Malcolm  Baldrige  National  Quality  Award  as  the  baseline  for  the  quality  award 
programs.  We  examined  the  process,  auditor  requirements,  and  results.  Chapters  II  and 
TTT  contain  the  analysis  of  these  programs.  In  the  course  of  our  study,  we  found  that  some 
elements  of  past  performance  programs,  such  as  their  information  collection 
requirements,  could  pertain  to  our  consideration  of  reciprocal  agreements  and  decided  to 
examine  some  past  performance  programs  as  well.  Past  performance  programs  are 
described  in  Chapter  IV.  We  reviewed  a  report  on  the  feasibility  of  a  common  quality 
certification  system  within  the  precision  gear  industry  and  contacted  existing  industry 
organizations  fostering  reciprocal  agreements.  Descriptions  of  the  report  and  the 
reciprocal  organizations  are  in  Chapter  V.  Current  DoD  policy  is  also  outlined  in  Chapter 
V.  Chapter  VI  concludes  the  report  with  summary  recommendations. 

STUDY  RESULTS 

There  are  many  reasons  why  a  supplier  would  want  to  win  an  award  or  become 
certified  or  registered  to  a  standard.  Foremost  is  the  prospect  of  increased  business 
opportunities.  Most  qualification  programs  work  as  thresholds — either  the  supplier  meets 
the  standard  of  the  program  required  by  the  customer  or  the  supplier  doesn’t  do  business 
with  that  customer.  The  prestige  that  comes  with  winning  a  quality  award  also  usually 
results  in  new  business  in  the  commercial  sector.  For  defense  contracts,  audits  by  DCMC 
to  determine  that  the  supplier  meets  an  ISO-like  system  tailored  to  the  needs  of  the 
contract  are  required.  New  acquisition  rules  now  let  DoD  buying  authorities  consider  the 
past  performance  of  a  contractor — quality,  schedule,  and  cost  performance — ^in  source 
selection. 

The  problems  with  audits  for  a  supplier  come  when  different  customers  require 
different  types  of  audits  under  different  types  of  certification/registration/accreditation 
programs.  Many  of  these  audits  are  similar  in  nature  and  content,  as  illustrated  by  our 
comparison  tables  in  Chapters  n  and  HI.  Similarity  among  programs  is  a  result  of  using 
the  20  elements  of  the  ISO  9000  standards  as  their  basis.  But  each  audit  costs  the 
supplier  and  the  customer  substantial  amounts  of  time  and  money.  Data  we  collected 
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shows  that  ISO  9000  registration  alone  can  cost  a  company  as  much  as  $30,000  initially 
and  up  to  $5,000  every  6  months  for  reassessment  to  maintain  the  registration.  ^ 

Shared  data  bases  and  reciprocal  agreements  among  companies  in  formal 
organizations  such  as  the  Coordinating  Agency  for  Supplier  Evaluations  (C.A.S.E.)  and 
the  National  Aerospace  and  Defense  Contractors  Accreditation  Program  (NADCAP)  have 
saved  their  member  companies  both  time  and  money.  One  C.A.S.E.-member  company 
reports  that  it  saves  an  average  of  100  audit  surveys  per  year.  Each  survey  entails  about 
32  hours  of  labor  and  travel  cost,  which  works  out  on  average  to  $5,000  per  survey.  A 
large  prime  contractor  company  reports  net  savings  of  $1.6  million  per  year  due  to  its 
NADCAP  membership.  Another  prime  reports  savings  of  over  200  audits  per  year. 
NADCAP  supplier  companies  report  savings  of  about  40%  fewer  audits  per  year.  Their 
business  volume  has  increased  $750,000  to  $2  million. 

Despite  such  reported  savings,  trust  remains  an  issue  for  most  primes — they  prefer 
to  do  their  own  quality  system  audits  rather  than  rely  on  those  done  by  one  of  their 
competitors,  and  they  worry  about  the  legal  repercussions  of  sharing  their  audit  results  of 
suppliers  with  other  customers.  C.A.S.E.  members  determined  that  their  organization 
presents  no  legal  liability,  as  demonstrated  by  the  fact  that  it  has  operated  for  over  30 
years  without  successful  legal  challenges.  The  trick  is  to  have  an  arrangement  whereby 
the  participating  original  equipment  manufacturers  (OEMs)  agree  on  the  processes  for  the 
audits,  the  auditor  qualifications,  and  the  criteria  for  the  audits  (usually  based  on  ISO- 
9000  with  some  sector-specific  additions).  Then  the  contractors  can  share  nonprejudicial 
data  on  the  process,  completion,  and  pass-fail  results  of  the  audit  without  worrying  about 
legal  repercussions  of  sharing  the  actual  audit  results.  This  is  the  procedure  C.A.S.E. 
uses  and  it  has  been  lawsuit-free  since  its  inception  in  1964.  C.A.S.E.  was  formed  when 
several  prime  contractors,  who  shared  many  of  the  same  suppliers,  banded  together  to 
derive  a  process  whereby  they  could  reduce  the  number  of  audits  and  assessments  they 
had  to  perform  on  their  suppliers. 

DCMC  encourages  and  benefits  from  the  audit  work  of  industry  organizations 
such  as  C.A.S.E.  and  NADCAP  because  DCMC  can  often  rely  on  their  audit  reports, 
thereby  reducing  or  eliminating  the  need  for  a  DCMC  audit.  To  date,  however, 
membership  in  these  organizations  is  not  yet  widespread  across  all  industry  sectors. 


^  Steven  M.  Terronez,  “The  Contractor  Performance  Certification  Program,”  Army  RD&A,  Sept-Oct 
1997. 
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DCMC  manages  more  than  360,000  prime  contracts  worth  more  than  $900  billion  at 
more  than  23,000  contractors  throughout  the  world.  Any  substantial  gains  to  the 
government  from  a  relationship  with  these  industry  organizations  is,  therefore,  hindered 
by  their  currently  limited  size  and  scope. 

RECOMMENDATIONS 

To  reduce  the  time  and  cost  of  quality  system  audits,  DoD  would  do  well  to  make 
use  of  as  much  industry  data  and  as  many  industry  programs  as  possible.  This  has  been 
DCMC  policy  since  November  1996,  and  we  recommend  that  DoD  continue  along  this 
path.  Because  DoD  cannot  recommend  one  industry  program  over  another,  industry  must 
make  the  determination  of  which  reciprocal  organization  meets  its  needs.  Since  trust  is 
best  built  among  customers  who  share  similar  products,  these  organizations,  or  divisions 
and  sections  within  the  organizations,  should  be  built  around  industry  sectors,  as  the 
sections  of  C.A.S.E.  and  NADCAP  are.  Since  DCMC  deals  with  thousands  of  products 
that  do  not  currently  have  reciprocal  agreements  and  organizations  within  their  sectors,  its 
only  option  is  to  do  its  own  audits  under  the  current  policy  of  using  any  other  audit 
information  available  from  the  supplier.  DoD  would  do  well  to  encourage  the  defense 
and  commercial  industry  to  look  at  the  benefits  of  forming  reciprocal  relationships  for 
their  quality  audits.  When  such  arrangements  become  best  commercial  practice,  DoD  can 
make  full  use  of  them.  As  with  other  processes  under  the  Single  Process  Initiative,  the 
greatest  benefit  comes  when  streamlining  practices  not  only  affect  prime  contractors  but 
also  pervade  the  supplier  base.  We  repeat  here  the  recommendations  of  the  gear  industry 
report  on  the  Common  Quality  Certification  System  (CQCS): 

Finally,  for  a  CQCS  to  be  truly  successful,  it  must  be  applicable  at  all 
supplier  levels.  Therefore,  primes  need  to  undertake  efforts  to  reduce  the 
redundancies  and  foster  reciprocity  in  the  auditing  and  certification  of  their 
own  suppliers.  Also,  the  primes  need  to  agree  on  criteria  and  encourage 
reciprocity  in  the  audit  and  certification  of  their  suppliers,  otherwise,  a 
CQCS  will  provide  inconsistent  benefits  to  the  industry  as  a  whole. 
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I.  INTRODUCTION 


Supplier  oversight  processes  entail  significant  expense  for  both  industry  and  DoD. 
Best  commercial  practice  seeks  to  reduce  this  expense  through  stringent  evaluations  of 
suppliers’  quality  system  effectiveness,  followed  by  reduced  oversight  and  long-term 
partnering.  Suppliers  that  meet  evaluation  criteria  gain  the  status  of  “preferred”  or 
“certified.” 

Currently,  companies  face  the  potential  that  their  product  and  process  quality  will 
undergo  numerous  audits  and  certifications  by  various  industry  bodies  as  well  as 
government  customers.  Audits  are  required  to  achieve  preferred  supplier  status  and 
Malcolm  Baldrige  finalist  status,  and  to  qualify  for  such  programs  as  the  Army’s 
Contractor  Performance  Certification  Program  (CP)2,  Boeing’s  D 1-9000  certification,  the 
Big  3  Automaker’s  QS-9000  certification,  and  ISO  9000  registration  and  qualification. 
But  because  there  is  no  universal  reciprocal  acceptance  among  the  various  industry 
associations  and  government  agencies,  the  savings  from  reduced  oversight  and  the 
elimination  of  redundant  audits  and  assessments  are  only  partially  realized  by  both  the 
customers  and  the  suppliers. 

A.  OBJECTIVE  AND  STATEMENT  OF  WORK 

IDA  was  tasked  to  determine  how  DoD  can  enter  into  reciprocal  agreements  with 
industry  quality  system  assessment  bodies  and  other  government  agencies  to  reduce 
DoD’s  own  oversight  resource  requirements.  A  particularly  noticeable  impact  is 
expected  on  the  subcontractor  tiers,  where  reciprocity,  spanning  primes  and  government 
buying  activities,  can  significantly  accelerate  the  Single  Process  Initiative  efforts  among 
subtiers. 

During  this  project  IDA  was  to — 

•  Develop  descriptions  and  comparisons  of  all  the  various  quality  system 
assessment  programs  and  determine  how  to  evaluate  equivalence.  Consider 
the  following  key  areas: 

-  Process  Management — ^how  is  the  process  controlled  to  be  consistent, 
stable,  and  respected? 
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-  Auditor  Qualification — what  are  the  training,  skill,  and  experience 
criteria  for  the  auditors? 

-  Assessment  Criteria — what  measures,  ratings,  or  maturity  indices  are 
used  to  give  a  passing  grade,  as  well  as  any  further  requirements  for  the 
retention  of  the  certification  or  registration  (periodic  reevaluations, 
continuous  improvement  plans,  etc.)? 

•  Analyze  the  options  for  entering  into  a  reciprocal  agreement,  influencing  the 
continued  validity  of  a  certification  program,  and  terminating  agreements  if 
the  program  fails  to  live  up  to  expectations. 

•  Discuss  how  DoD  can  increase  efficiency  through  reduced  oversight. 

B.  KEY  CONCEPTS 

Many  terms  used  throughout  this  paper  need  to  be  defined  and  explained  because 
their  use  varies,  even  within  the  quality  assurance  community.  In  general,  these  terms 
relate  to  customers’  acceptance  of  quality  system  evaluations  conducted  by  certification 
or  registration  bodies  as  a  means  to  streamline  the  resources  required  for  contract 
monitoring.  More  specifically,  they  indicate  the  type  of  acceptance,  means  of  acceptance, 
and  degree  of  acceptance. 

1.  Types  of  Acceptance 

The  term  certified  usually  applies  to  a  product  or  service  of  a  supplier  that  is 
found  to  be  in  compliance  with  a  certain  standard.  In  her  paper  The  ABC’s  of 
Certification  Activities  in  the  United  States,  Maureen  Breitenberg  of  the  National  Institute 
of  Standards  and  Technology  (NIST)  defines  a  certification  program  or  scheme  as  “the 
procedure  by  which  written  assurance  is  given  that  a  product  or  service  conforms  to  a 
standard  or  specification.”  She  concedes,  however,  that  many  other  terms  exist  and  are  in 
use  by  various  organizations  to  define  the  same  thing. 

In  the  same  vein,  Breitenberg  says,  quality  system  registration,  which  involves 
the  assessment  and  periodic  audit  of  the  effectiveness  of  a  supplier’s  quality  system,  is 
often  misnamed  quality  system  certification.  A  supplier  can  register  a  quality  system 
with  an  organization  that  conducts  a  registration  program  without  participating  in  a 
product  certification  program. 

Certification/registration  bodies  exist  for  all  kinds  of  processes  and  systems.  One 
data  source,  a  NIST  publication,  is  a  compendium  of  industrial  certification  bodies.  This 
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report  concentrates  on  quality  system  registration,  although  reference  is  made  to  other 
processes  for  certification. 

The  Defense  Contract  Management  Command  (DCMC)  uses  the  term  qualified  to 
mean  that  the  contractor  satisfies  the  appropriate  elements  of  ISO  9000  but  may  not  be 
registered  (or  certified)  to  the  standard. 

2.  Means  of  Acceptance 

First  party  audit — ^an  audit  that  an  organization  conducts  on  its  own  systems. 
These  self-audits  are  performed  by  the  organization’s  own  staff. 

Second  party  audit — an  audit  that  one  organization  performs  on  another  with 
which  the  auditing  organization  either  has,  or  intends  to  enter  into,  a  contract  to  purchase 
goods  or  services.  These  audits  can  be  carried  out  by  the  purchasing  organization  or  an 
outside  agency  under  contract  or  agreement  with  the  purchasing  organization. 

Third  party  audit — an  audit  that  a  third  party  agency  (a  body  not  controlled  by  or 
under  the  influence  of  the  customer  or  supplier)  performs  to  determine  whether  a  product 
or  service  complies  with  an  industrial,  national,  or  international  standard.  The  audit 
could,  in  turn,  be  used  to  assure  current  and  prospective  customers  of  the  product  or 
service. 

Third  party  audits  bring  up  the  issue  of  who  will  approve  the  registrar.  The 
American  Society  for  (Quality  (ASQ)*  addressed  this  problem  by  instituting  the 
Registration  Accreditation  Board  (RAB)  to  certify  auditors  and  accredit  registrars  to 
conduct  ISO  9000  and  ANSI/ASQC  Q9000  registrations.  RAB  subsequently  became 
independent  of  ASQ  and  joined  with  the  American  National  Standards  Institute  (ANSI) 
to  operate  the  U.S.  National  Accreditation  Program  (NAP)  for  the  accreditation  of 
registrars.  RAB  independently  certifies  auditors. 

3.  Degree  of  Acceptance 

Recognition^  is  a  concept  whereby  a  potential  customer  obtains  and  gives 
credence  to  the  results  of  second  or  third  party  audits  and  assessments.  With  some 


1  The  American  Society  for  Quality  (ASQ)  was  formerly  known  as  the  American  Society  for  Quality 
Control  (ASQC). 

^  Recognition  is  used  here  to  mean  “acknowledgement  or  special  notice  and  attention,”  not  the  formal 
legal  sense  of  international  standardization  recognition  arrangements. 
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understanding  of  how  the  results  were  gathered  and  how  the  auditor  used  them,  a 
potential  customer  may  be  able  to  forgo  a  full  set  of  new  audits.  But  the  supplier  must  be 
willing  to  offer  the  information  and  the  customer  willing  to  give  it  credence.  How  much 
credence  depends  on  how  well  the  information  satisfies  the  customer’s  needs. 

Reciprocity  is  a  concept  whereby  a  pair  or  group  of  potential  customers  agree  to 
completely  accept  one  another’s  or  third  party  audits.  That  is,  the  customer  accepts  not 
only  the  audit  results,  but  also  the  process  by  which  they  were  obtained,  including  the 
qualifications  of  the  auditors  performing  the  audit.  For  example,  if  the  Army’s 
Contractor  Performance  Certification  Program  (CP)2  and  the  automakers’  QS-9000 
program  had  a  reciprocal  agreement,  they  would  accept  each  other’s  supplier  registration. 
In  this  instance,  the  supplier’s  participation  in  QS-9000  and  registered  status  offer  a 
streamlining  opportunity  to  the  Army  (CP)2  program  rather  than  merely  the  products  of  a 
single  audit.  One  can  see  these  types  of  agreements  are  important  in  dealing  with 
products  that  have  to  be  standardized  for  interoperable  use. 

Reciprocity  can  be  exercised  bilaterally  or  multilaterally,  but  multilateral 
agreements  offer  a  greater,  long-term  opportunity  for  efficiency — all  parties  are  then 
motivated  to  operate  in  a  mutually  beneficial  manner.  For  example,  mutual  recognition 
by  several  primes  of  each  other’s  audit  results  can  greatly  reduce  audits  of  subcontractors. 
Memorandums  of  understanding  between  the  automobile  and  the  aerospace  programs 
would  help  ensure  that  the  programs  remain  synchronized  to  the  maximum  extent 
possible,  and  that  all  common  suppliers  only  have  to  work  with  a  single  process.  Such 
bilateral  and  multilateral  agreements  are  uncommon,  however.  Nevertheless,  decisions 
by  customers  to  accept  certification  results  from  two  or  more  programs  can  work  to  great 
advantage.  The  basis  for  making  the  decisions  is  the  same. 

C.  EXAMPLES  OF  RECIPROCITY 

Reciprocal  certification  schemes  are  often  used  between  countries  or  states.  The 
European  Union  (EU),  for  example,  has  established  a  mutual  recognition  approach  for 
accepting  certifications  of  regulated  products  that  is  in  line  with  its  goal  of  identifying 
approaches  to  the  “technical  harmonization  of  standards.”  Each  EU  member  state 
determines  who  is  competent  to  perform  conformity  assessments  and  presents  a  list  of 
certifiers  to  the  EU  Commission.  EU  members  are  required  to  determine  that  the 
certifiers  recommended  for  inclusion  on  the  EU  Commission’s  certifier  list  meet  the 
criteria  contained  in  the  EN  45000  series.  These  certifiers  are  called  upon  to  assess  and 
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certify  that  products  meet  established  requirement  criteria.  Once  a  listed  certifier 
adjudges  a  product  acceptable,  whether  through  certification  or  other  approval  marking, 
the  product  is  to  be  considered  adequately  certified  by  all  EU  member  countries.^ 

Certifications  with  varying  degrees  of  reciprocity  also  apply  to  lawyers,  teachers, 
physician  organizations,  and  women  business  owners.  Examples  of  their  recognition  and 
reciprocity  experiences  are  as  follows.  Descriptions  are  summarized  from  source 
material  listed  in  Appendix  D. 

Lawyers.  At  present,  members  of  the  legal  profession  remain  certified  on  the 
local  or  state  level.  Each  of  these  jurisdictions  administers  its  own  comprehensive  exam 
that  individuals  must  pass  in  order  to  practice  law  in  that  state.  Successful  completion  of 
this  exam  certifies  an  individual’s  competency  for  legal  general  practice.  Increased 
mobility  of  the  society  and  trends  toward  increased  specialization  in  the  workforce  have 
led  some  in  the  legal  field  to  advocate  developing  a  national  legal  certification  series  to 
certify  that  individuals  who  pass  an  exam  or  exams  are  proficient  in  general  law  and/or 
additional  legal  specializations,  regardless  of  state  jurisdictions. 

Teachers.  Teachers  in  the  United  States  are  certified  at  the  state  level,  but 
numerous  interstate  certification  agreements  exist  to  establish  the  process  that  a  certified 
teacher  must  complete  to  become  certified  in  a  different  state.  These  agreements  exist 
mostly  to  accommodate  teachers  moving  from  one  state  to  another  and  wishing  to  acquire 
a  teacher  certification  in  their  new  state  of  residence.  If  two  states  have  an  interstate 
certification  arrangement,  a  certified  teacher’s  educational  training  in  one  state  will  be 
recognized  in  the  other  member  state.  Even  though  an  interstate  certification  agreement 
exists,  however,  it  may  not  necessarily  waive  any  additional  tests  or  non-educational 
qualifications  required  by  a  particular  state  for  a  teacher’s  certification.  The  interstate 
certification  agreements,  therefore,  establish  the  framework  for  recognition  of  teacher’s 
educational  preparation  from  state  to  state.  A  true  reciprocal  arrangement  with  respect  to 
teacher  certifications  would  require  that  a  state  accept  a  teacher’s  certification  firom 
another  state  without  any  additional  qualifications. 

Physicians  Organizations.  The  National  Committee  for  Quality  Assurance 
(NCQA)  is  a  private,  not-for-profit  organization  that  evaluates  medical  managed  care 
plans  on  the  basis  of  quality.  Since  1991,  NCQA  has  been  accrediting  managed  care 


3  Maureen  A.  Breitenberg,  “The  ABC’s  of  Certification  Activities  in  the  United  States,”  April  1997. 
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plans  on  the  basis  of  their  quality  improvements,  physicians’  credentials,  members’  rights 
and  responsibilities,  preventive  health  services,  utilization  management,  and  medical 
reports.  Although  this  accreditation  is  performed  on  a  voluntary  basis,  half  of  HMOs 
have  a  relationship  with  NCQA,  and  more  than  75  percent  of  Americans  who  belong  to 
HMOs  are  in  ones  that  have  been  assessed  by  NCQA.  In  addition,  the  Physician’s 
Organization  Certification  Program  was  introduced  in  October  1997  with  the  goal  of 
decreasing  the  number  of  audits  of  physician’s  groups  by  health  plans  for  which  they  are 
under  contract.  Audit  teams  composed  of  physicians  and  managed  care  experts  evaluate 
physician’s  organizations  and  provide  their  findings  to  an  NCQA  national  oversight 
committee  for  the  final  decision  on  certification.  The  Physician’s  Organization 
Certification  has  replaced  the  multiple  audits  that  managed  care  organizations  were  each 
required  to  perform  on  their  contracting  physician’s  organizations  in  order  to  assure 
appropriate  oversight. 

Women  Business  Owners.  Businesses  hoping  to  gain  a  competitive  advantage  in 
the  acquisition  process  because  they  are  owned  or  controlled  by  a  woman  have 
traditionally  faced  numerous  and  redundant  certifications  of  their  ownership  status  by 
corporations  and  government  agencies  as  they  compete  for  contracts.  In  January  1997, 
the  Women  Business  Owners  Corporation  (WBOC)  introduced  its  National  Certification 
Program  with  the  goal  of  creating  one  certification  process  to  validate  a  business’  woman 
owned  status  and  thus  increase  the  competitiveness  of  woman  owned  businesses.  The 
WBOC  Consortium  members  assess  applicants  and  certify  that  a  business  is  owned  or 
controlled  by  a  woman.  In  addition,  the  National  Certification  Program  provides  for 
reciprocity,  in  that  the  determination  of  a  business’  woman  owned  status  by  one  WBOC 
Consortium  member  is  to  be  accepted  in  its  entirety  by  the  other  WBOC  Consortium 
members. 

To  become  certified,  an  applicant  submits  a  certiHcation  application  and 
additional  third  party  analyses  that  document  the  business’  legal  status,  financial 
background,  and  appraisals  to  the  Consortium  member  in  whose  jurisdiction  the 
applicant’s  headquarters  is  located.  The  Consortium  member’s  Certificate  Committee 
reviews  the  application  and  third  party  analyses  and  conducts  an  on-site  audit,  including 
interviews,  scheduled  and  unscheduled  visits,  and  the  evaluation  of  other  documentary 
material.  The  Certificate  Committee  then  makes  recommendations  to  the  Consortium 
member’s  Board  of  Directors,  which  performs  a  final  review  of  the  businesses’ 
application,  third  party  analyses,  and  site  visit  reports  and  grants  or  denies  certification 
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that  a  business  is  woman  owned  or  controlled.  Those  businesses  that  receive  certification 
are  added  to  the  WBOC  International  Women  Business  Owners  Database,  which 
government  and  industry  may  access  when  considering  businesses  for  a  contract. 

D.  INDUSTRY  BEST  PRACTICE  FOR  SUPPLIER  CERTIFICATION 

Recently,  supplier  certification  or  preferred  supplier  programs  have  become 
popular  in  industry.  Juran  defines  supplier  certification  as  “the  process  of  evaluating  the 
performance  of  a  supplier,  with  respect  to  product  quality,  with  the  view  of  authorizing 
the  supplier  to  self-certify  shipments  (if  the  evaluation  concludes  that  the  supplier’s 
performance  justifies  such  authorization).”^  Supplier  certification  programs  are  prevalent 
among  prime  contractors.  In  this  section,  we  briefly  describe  specific  prime  contractor 
supplier  certification  programs  to  illustrate  some  of  the  benefits  to  both  the  customer  and 
the  supplier.  Descriptions  are  summarized  from  source  material  listed  in  Appendix  D. 

In  order  to  qualify  for  certification,  suppliers  of  Raytheon  Missile  Systems  must 
have  a  proven  track  record  for  product  quality  and  on-time  deliveries.  Once  certified, 
suppliers  are  able  to  ship  their  products  to  Raytheon  facilities  with  minimal  inspections. 
Some  of  the  claimed  benefits  of  this  program  are  improved  quality,  increased 
productivity,  elimination  of  material  returns,  and  a  reduction  in  on-site  inspection. 

Texas  Instruments  DS&EG  has  a  formal  supplier  certification  program  in  which 
the  supplier  is  primarily  responsible  for  incoming  quality.  TI  conducts  statistical  process 
control  (SPC)  training  at  the  supplier’s  facility  and  establishes  the  performance  criteria 
necessary  for  certification.  Once  certified,  suppliers  receive  preferential  consideration 
from  TI  for  new  or  follow-on  work.  The  claimed  benefits  of  this  program  include  the 
reduction  of  redundant  tests  and  inspection  and  improvements  in  measurement 
techniques. 

Lockheed  Martin  has  a  program  in  which  suppliers  can  achieve  three  different 
levels  of  certification — ^bronze,  silver,  and  gold.  Bronze  certification  requires  a  1(X) 
percent  quality  part  rating,  a  minimum  of  six  lots  delivered,  and  an  approved  quality 
system.  The  silver  level  requires  an  SPC  program,  a  management  letter  of  intent,  and  an 
on-site  review.  To  receive  the  Gold  level  certification,  the  supplier  must  also  have  a 
continuous  process  improvement  program.  When  certified,  suppliers  benefit  firom 


^  J.M.  Juran  and  Frank  M.  Gryna,  Juran’ s  Quality  Control  Handbook,  Fourth  Edition,  McGraw-Hill, 
NY. 
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reduced  surveillance  and  a  preferred  procurement  status.  Lockheed  Martin  GES  claims 
many  benefits  from  this  program,  including  the  following: 

•  A  2. 1  percent  improvement  in  purchased  material  inspection  yield 

•  A  67  percent  reduction  in  back  lot  backlog 

•  A  55  percent  reduction  in  cycle  time 

•  An  87  percent  reduction  in  scrap. 

AlliedSignal  has  two  approaches  to  its  objective  of  total  product  quality  and  a 
commitment  to  excellence.  First,  through  its  technology  excellence  endeavors, 
AlliedSignal  has  focused  on  management  and  processes  in  an  effort  to  improve  product 
quality.  The  company  has  introduced  product  scorecards  as  a  means  of  tracking  the 
present  and  goal  sigmas  for  parts.  In  particular,  variations  are  identified  for  deficient 
design  margins,  substandard  incoming  parts  and  material,  and  inadequate  process 
capabilities.  By  introducing  such  methods,  AlliedSignal  is  expected  to  increase  quality, 
reduce  cycle  times,  and  better  satisfy  its  customers.  Second,  and  more  directly  related  to 
the  certification  of  suppliers,  AlliedSignal  has  implemented  an  Integrated  Product 
Development  System  (IPDS)  that  addresses  such  issues  as  lifecycle  acquisition, 
collaboration  across  traditional  functional  areas,  and  the  sharing  of  accountability.  ISO 
9001  is  used  by  IPDS  teams  as  a  reference  checklist  to  determine  supplier  compliance. 

With  the  current  industry  focus  on  maximizing  value  and  quality  while  also 
determining  ways  to  eliminate  the  redundancy  of  basic  systems  audits,  British  Aerospace 
has  introduced  its  Preferred  Supplier  Process  (PSP).  Through  this  process,  a  supplier  is 
evaluated  on  the  basis  of  three  general  categories:  1)  a  business  assessment,  2)  statistical 
process  control,  and  3)  performance.  Within  these  criteria,  written  information  or  metric 
data  is  collected  on  each  supplier  for  items  such  as  customer  satisfaction,  quality  policies 
and  methods,  the  prevention  of  problems,  delivery,  technology,  support,  statistical 
process  control  plans  and  audits,  and  responsiveness.  Given  the  results  of  a  supplier’s 
evaluation  based  on  these  categories  and  measures,  an  award  agreement,  which 
emphasizes  the  continuous  improvement  of  performance,  is  created  between  the  supplier 
and  British  Aerospace.  In  the  end,  the  supplier  is  assigned  a  gold,  silver,  or  bronze 
supplier  award  status. 

The  Northrop  Grumman  Commercial  Aircraft  Division  (NGCAD)  has  developed 
an  Advanced  Quality  Practices  (AQP)  program  as  a  means  of  preventing  supplier  defects 
and  thus  reducing  the  amount  of  variability  witnessed  in  the  products  and  services  that  it 
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receives  from  its  suppliers.  Suppliers  use  an  AQP  assessment  survey  to  determine  how 
their  products  and  processes  match  up  with  the  following  criteria:  1)  management 
commitment,  2)  procedures  and  documentation,  3)  training,  4)  key  characteristics,  5) 
improvement  tools,  6)  control  charting,  7)  key  characteristic  data  analysis,  8)  process 
improvement’,  and  9)  AQP  flowdown.  NGCAD  conducts  on-site  assessments  to  ensure 
that  the  necessary  elements  are  in  place  to  maintain  key  characteristics  and  continued 
improvement.  Any  identified  deficiencies  and  their  related  corrective  actions  are  also 
reviewed  by  NGCAD,  and  additional  surveillance  assessments  of  a  supplier  are 
conducted  on  an  as  needed  basis. 

In  addition  to  these  specific  prime  contractor  supplier  certification  programs,  one 
industry  has  proposed  a  new  approach  for  companies  to  obtain  quality  management 
system  (QMS)  registration.  In  1994,  the  electronics  industry,  led  by  Hewlett-Packard  and 
Motorola,  introduced  an  alternative  method  whereby  a  company  could  pursue  and 
maintain  its  QMS  registration.  This  effort  was  launched  in  response  to  the  high  cost  of 
third  party  certification  and  the  belief  that  certification  requirements  were  becoming  a 
new  type  of  trade  barrier  (the  electronics  industry  is  highly  dependent  upon  trade).  Their 
philosophy  is  reflected  in  the  following  quote. 

[A]  company  that  has  already  achieved  and  demonstrated  an  effective, 
high-performing  quality  system  should  be  allowed  greater  flexibility  [and] 
less  cost  and  bureaucracy  to  obtain  or  maintain  accredited  third-party 
certificates  to  management  system  standards.^ 

The  Supplier  Audit  Confirmation  (SAC)  or  System-Level  Assessment’s  approach, 
therefore,  provides  companies  with  more  autonomy  in  their  QMS  registration  and 
replaces  multiple  audits  with  one  that  is  performed  by  both  internal  and  external  auditors. 

The  SAC  method  envisions  that  a  company  would  be  able  to  prove  and  maintain 
its  ISO  9000  registration  through  the  documented  results  of  both  third  party  and  internal 
audits.  Third  party  audits  would  be  performed  only  for  three  ISO  9000  criteria 
(management  review  system,  internal  audit  process,  and  corrective  action  processes), 
while  the  company’s  internal  audits  would  be  accepted  in  determining  compliance  to  the 
remaining  17  criteria.^  A  company’s  internal  auditors  would  have  to  meet  specified 
qualifications  relating  to  their  knowledge  of  quality  and  auditing  experience.  The  third 


5  “Questions  and  Answers,”  System-Level  Assessment  (formerly  called  SAC)  home  page. 
^  See  Chapter  II  for  a  discussion  of  ISO  9000  and  its  criteria. 
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party  would  also  retain  the  right  to  review  and  test  a  company’s  internal  audit  processes 
in  order  to  verify  the  competence  of  the  internal  auditors. 

Proponents  of  the  SAC  admit  that  this  approach  may  be  suitable  only  for  those 
companies  and/or  industries  that  already  possess  advanced  quality  systems.  Any 
company  using  this  system,  however,  can  reduce  its  costs,  especially  a  company  with 
multiple  facility  sites.  Under  the  SAC  system,  a  company’s  internal  audits  that 
encompass  all  facility  sites  can  be  considered  in  place  of  the  individual  third  party 
auditing  of  each  site. 

E.  OUTLINE  OF  THE  REPORT 

Chapters  n  and  III  describe  and  compare  several  quality  system  assessment 
programs  and  quality  award  programs.  For  purposes  of  this  task,  we  consider  the 
baseline  for  quality  system  audits  to  be  registration  of  a  contractor’s  quality  management 
system  to  ISO  9000  or  its  U.S.  equivalent,  ANSI/ASQ  Q9000.  The  baseline  for  quality 
awards  is  the  Malcolm  Baldrige  National  Quality  Award.  Because  current  government 
policy  includes  past  performance  as  a  source  selection  criterion,  many  government 
programs  collecting  information  on  contractor  past  performance  offer  some  elements 
useful  in  the  consideration  of  reciprocal  agreements  for  quality  system  assessments.  Past 
performance  programs  are  described  in  Chapter  IV.  Reciprocal  agreement  organizations 
and  policy  are  discussed  in  Chapter  V.  Chapter  VI  concludes  the  report  with  summary 
reconunendations. 
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n.  QUALITY  SYSTEM  ASSESSMENT  PROGRAMS 


This  chapter  summarizes  and  compares  the  current  quality  system  assessment 
programs  in  industry  and  the  government,  using  ISO  9000  (or  its  U.S.  equivalent, 
ANSI/ASQ  Q9000)  as  a  baseline  for  comparison.  Most  of  the  programs  in  this  chapter  are 
based  on  ISO  9000  and  in  fact  expand  on  its  qualifications.  The  program  descriptions  are 
summarized  from  the  specific  references  listed  in  Appendix  D.  A  comparison  of  the 
specific  criteria  of  the  different  programs  is  included  at  the  end  of  the  chapter,  and  the 
specific  criteria  are  listed  in  Appendix  A. 

A.  BASELINE— ISO  9000 

With  the  increasing  flow  of  goods  and  services  from  one  country  to  another  in  the 
global  marketplace,  a  need  developed  for  a  system  of  guidelines  to  standardize  varying 
national  approaches  to  quality.  Recognizing  that  need,  the  International  Organization  for 
Standardization  (ISO)  established  ISO  9000*  as  a  system  of  procedural  guidelines  to  be 
implemented  and  monitored  by  companies  to  ensure  the  consistency  of  their  products  and 
services.  The  purpose  of  ISO  9000  is  twofold:  1)  to  explain  the  similarities  and 
dissimilarities  of  different  quality  concepts;  and  2)  to  provide  advice  on  how  to  choose  and 
implement  these  established  international  standards  as  a  means  of  achieving  internal 
quality  management. 

ISO  9000,  in  actuality,  represents  a  series  of  standards  that  includes  IS09000-1, 
ISO9001,  IS09002,  ISO  9003,  and  IS09004-1.  Each  of  the  ISO  9000  standards  provides 
a  quality  assurance  model  to  be  used  by  companies,  but  each  deals  with  a  separate 
functional  area.  The  functional  focus  areas  are  broken  down  as  follows:  design, 
development,  production,  installation  and  servicing  (ISO9001);  production  and  installation 
and  servicing  (IS09002);  final  inspection  and  test  (IS09003);  and  quality  management 
and  quality  systems  elements  (IS09004-1).  All  of  the  standards  represented  by  the  ISO 
9000  series  illustrate  the  types  of  elements  that  should  exist  in  a  company’s  quality  system 
but  do  not  prescribe  how  to  implement  them.  Individual  companies  have  their  own 


*  The  U.S.  equivalent  standard  is  the  American  Society  for  Quality  (ASQ)  Q9000  series  of  standards. 
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specific  products,  objectives,  and  requirements  and  therefore  must  design  and  implement 
processes  that  satisfy  the  general  guidelines  as  well  as  their  own  unique  needs. 

ISO  standards  are  expected  to  go  through  a  process  of  revision  every  5  years.  Such 
an  endeavor  is  currently  being  conducted  by  the  ISO  technical  committee  (TC)  176,  which 
is  the  body  responsible  for  ISO  standards.  This  revision  is  tentatively  due  in  draft  form  in 
mid  1999.  The  focus  of  TC  176  has  been  to  listen  to  the  input  of  ISO  9000  customers  and 
seek  the  convergence  of  quality  standards,  both  within  and  outside  the  ISO  9000  family  of 
standards.  As  a  start,  TC  176  is  expected  to  discontinue  the  present  IS09002  and 
IS09003  as  independent  entities  and  integrate  them  into  ISO9001.  In  addition,  in 
reference  to  standards  outside  the  ISO  9000  family,  the  TC  176  is  looking  at  ways  to 
address  specific  industry  issues  within  the  revisions.  In  fact,  it  has  considered  making  the 
enhanced  basic  portion  of  QS-90002  a  part  of  the  ISO9001  revision. 

From  this  point  forward  in  this  report,  however,  the  term  ISO  9000  will  be 
considered  to  refer  specifically  to  ISO9001,  which  focuses  on  the  processes  of  design, 
development,  production,  installation  and  servicing.  ISO  9000  most  often  is  associated 
with  ISO9001  since,  of  the  four  quality  assurance  models  in  the  ISO  9000  standard  series, 
ISO9001  is  the  most  comprehensive  in  scope. 

Companies  seeking  ISO  9000  registration  are  evaluated  on  the  basis  of  20  criteria: 
1)  management  responsibility;  2)  quality  system;  3)  contract  review;  4)  design  control;  5) 
document  and  data  control;  6)  purchasing;  7)  control  of  customer-supplied  product;  8) 
product  identification  and  traceability;  9)  process  control;  10)  inspection  and  testing;  1 1) 
control  of  inspection,  measuring  and  test  equipment;  12)  inspection  and  test  status;  13) 
control  of  nonconforming  product;  14)  corrective  and  preventive  action;  15)  handling, 
storage,  packaging,  preservation,  and  delivery;  16)  control  of  quality  records;  17)  internal 
quality  audits;  18)  training;  19)  servicing;  and  20)  statistical  techniques.  Through  the  ISO 
9000  registration  process,  a  third  party  provides  assmance  that  a  company  or  organization 
has  fulfilled  quality  management  system  requirements  of  the  standard. 

The  time  to  attain  ISO  9000  registration  will  vary,  ranging  fi-om  6  to  18  months 
depending  upon  the  size  of  the  company  seeking  registration  and  the  initial  maturity  of  its 
quality  system.  During  the  preassessment  phase,  the  company  may  perform  a  self- 
assessment  or  contract  with  a  third  party  registration  agency  to  assess  the  existing  quality 


2  See  Section  B.3  of  this  chapter  for  QS-9000. 
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management  system.  If  the  existing  system  does  not  meet  the  requirements,  a  quality 
system  that  addresses  ISO  9000  standards  and  criteria  is  then  implemented.  In  the  formal 
assessment  phase,  the  third  party  registration  agency  (registrar)  conducts  an  audit  of  the 
quality  management  system  to  determine  its  compliance  with  the  requirements  of  the  ISO 
9000  standard.  Once  the  registrar  determines  that  the  company  or  organization  is  in 
compliance  with  ISO  9(X)0  requirements,  the  registrar  issues  a  certificate  of  compliance. 
The  integrity  of  the  registration  is  upheld  through  the  use  of  surveillance  audits,  which  can 
be  undertaken  at  varying  intervals  depending  on  the  registrar.  The  registration  usually 
expires  at  the  end  of  3  years. 

B.  INDUSTRY  PROGRAMS 

1.  Boeing  Dl-9000 

In  the  aerospace  industry,  studies  have  shown  that  production  costs  are  adversely 
affected  by  the  cost  associated  with  correcting  the  quality  problems  of  parts  produced  with 
production  methodologies  based  on  cost  rather  than  quality.  Boeing  established  and 
implemented  the  quality  approval  system  Dl-9000  to  factor  in  the  cost  and  savings 
associated  with  ensuring  that  the  company,  its  suppliers,  and  subcontractors  conform  to 
certain  quality  standards.  The  overriding  objective  of  Dl-9000,  therefore,  is  to  provide  a 
supplier  with  the  tools  and  methods  necessary  to  conunit  to  ongoing  improvements  in  the 
quality  of  products  and  processes  that  will  yield  fewer  quality  defects,  reduced  waste, 
satisfied  customers,  and  continuing  profits. 

Dl-9000  encompasses  two  types  or  phases  of  approval:  1)  Basic  Quality  System 
(BQS)  and  2)  Advanced  Quality  System  (AQS).  The  Dl-9000  BQS  has  been  structured 
using  the  quality  standard  criteria  from  IS09002.  Although  these  1S09002  criteria  are 
considered  the  minimum  requirements  to  receive  Dl-9000  BQS  approval,  a  supplier’s 
third  party  ISO  registration  does  not  automatically  translate  into  Boeing  approval  of  that 
supplier.  The  AQS  approval  represents  a  process  whereby  a  supplier  meets  the  criteria  for 
the  BQS  and  can  prove  its  ability  to  maintain  and/or  reduce  the  variations  in  key 
characteristics  (which  are  the  vital  hardware  or  process  factors  that  have  a 
disproportionately  high  effect  on  the  performance  of  the  end  product)  in  order  to  improve 
quality.  For  both  types  of  approval,  D 1-9000’s  approach  toward  the  20  criteria  and  key 
characteristics  focuses  on  those  products  and  services  directly  related  to  a  supplier’s 
relationship  with  Boeing. 
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Boeing  suppliers  must  achieve  D 1-9000  BQS  or  AQS  approval  and  are  encouraged 
by  Boeing’s  procurement  quality  assurance  representatives  to  adopt  the 
D 1-9000  quality  approach.  During  the  formal  assessment  phase,  a  supplier’s  quality  and 
production  processes  are  studied.  Suppliers  then  respond  to  any  nonconfomuties  with 
D 1-9000  criteria  that  were  unveiled  during  the  initial  audit,  and  additional  audits  are 
conducted  to  ensure  that  appropriate  corrective  actions  have  eliminated  the  previously 
noted  deficiencies.  Once  the  supplier  satisfactorily  implements  all  corrective  actions,  it 
receives  D 1-9000  approval.  During  the  postassessment  phase,  a  supplier  maintains  its 
Boeing  D 1-9000  approval  status  by  submitting  further  maintenance  audits,  while  Boeing 
monitors  the  report  of  any  quality  violations  or  concerns  raised  by  the  supplier’s 
customers.  Furthermore,  a  Boeing  Dl-9000  supplier  is  expected  to  flowdown  these 
quality  approaches  to  its  subcontractors,  thereby  improving  quality  throughout  all 
processes  contributing  to  the  production  of  products  or  services. 

2.  Aerospace  Industry  AS9000 

The  Aerospace  Basic  Quality  System  Standard  (AS9000)  was  created  by  a  group 
of  prime  contractors  from  the  aerospace  industry  acting  as  a  subcommittee  under  the 
auspices  of  the  Aerospace  and  Defense  Division  of  the  ASQ.  AS90(X)  was  developed  in 
an  effort  to  eliminate  redundant  and  overlapping  quality  requirements  in  the  aerospace 
industry,  without  jeopardizing  the  continuing  production  of  quality  products  and  services. 

The  overriding  objective  of  AS90(X)  is  to  focus  on  processes  in  order  to  provide  for 
an  ongoing  reduction  in  defects.  Indeed,  the  AS9000  philosophy  is  not  only  to  identify 
defects,  but  to  seek  out  and  implement  the  necessary  means  for  preventing  defects. 

AS9000  was  developed  as  an  aerospace  industry  quality  standard  because  ISO 
9000  was  regarded  as  inadequate  to  ensure  quality  within  that  industry.  The  subconunittee 
that  drafted  AS9000  did  take  ISO  9000  into  consideration,  however,  along  with  Boeing’s 
Dl-9000  and  other  existing  quality  standards.  AS9000  draws  its  basic  structure  from  the 
20  criteria  spelled  out  in  ISO  9000,  but  it  enhances  the  20  criteria  with  aerospace  industry- 
specific  requirements  and  a  series  of  notes  that  address  key  characteristics,  the  flowdown 
of  quality,  and  other  specific  industry  needs. 

A  supplier  that  is  seeking  AS9000  validation  presently  has  several  options.  The 
American  Aerospace  Quality  Group  (AAQG),  which  is  responsible  for  AS9000,  lists 
several  means  for  obtaining  validation  to  AS9000:  first  party  self-assessment,  second  party 
audits,  or  audits  by  a  third  party  Performance  Review  Institute  (PRI)  registrar.  The 
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AS9000  process  criteria  should  be  considered  as  compatible  with  those  implemented  for 
ISO  9000  registration.  Indeed,  the  AS9000  compliance  process  may  be  conducted  in 
tandem  with  a  supplier’s  ISO  9000  registration.  In  addition,  under  the  auspices  of  the 
National  Aerospace  and  Defense  Contractors  Accreditation  Program  (NADCAP),  a 
supplier  can  supplement  its  reaccreditations  and  ISO  surveillance  audits  with  AS9000 
coiiipliance.  ^ 

3.  Automakers  QS-9000 

In  1992,  the  three  major  automakers  (Ford,  Chrysler,  and  General  Motors) 
undertook  the  challenge  to  create  a  single  automotive  industry  quality  standard  to  replace 
their  individual  quality  requirements.  QS-9(X)0  was  the  resulting  automotive  industry 
quality  standard  bom  of  this  process  to  simplify  the  quality  standards  environment  in 
which  suppliers  to  the  Big  3  were  operating.  The  purpose  of  QS-9000  is  threefold:  1)  to 
decrease  the  number  of  audits  and  their  redundancies  as  experienced  by  suppliers;  2)  to 
provide  the  documentation  and  control  necessary  to  continuously  improve  quality;  and  3) 
to  improve  suppliers’  dialogue  with  their  customers. 

QS-9000  is  an  automotive  industry-specific  quality  system  based  on  the  framework 
of  ISO9001.  The  20  basic  criteria  detailed  in  ISO9001  are  considered  a  prerequisite  for 
QS-9000  registration;  however,  ISO9001  conformance  alone  may  be  insufficient  because 
it  does  not  include  the  additional  automotive  industry  and  customer  requirements  of  the 
QS-9000  criteria.  In  addition,  although  QS-9000  owes  its  basic  stracture  to  ISO  9000,  the 
two  standards  have  a  fundamental  difference  in  their  approaches  to  quality.  Whereas  ISO 
9000  is  a  descriptive  document  for  a  quality  system,  QS-9000  adopts  a  much  more 
prescriptive  approach,  detailing  not  only  what  a  quality  system  should  contain,  but  also 
how  it  may  be  achieved.  Furthermore,  QS-9000  approval  involves  satisfying  sector- 
specific  requirements  (production  part  approval  process,  continuous  improvement,  and 
manufacturing  capabilities)  and  those  requirements  specified  individually  by  Chrysler, 
Ford,  and  General  Motors. 

Internal  and  external  suppliers  of  the  Big  3  have  been  given  a  timetable  for  attaining 
QS-9000  registration.  The  QS-9000  registration  process  very  closely  mirrors  that  of  ISO 
9000.  The  supplier  begins  the  process  by  receiving  information  on  the  QS-9000  quality 
system.  During  the  presassessment  phase,  the  supplier  formally  applies  for  QS-9000 


^  See  further  reference  to  PRI  and  NADCAP  in  Section  V.B.2. 
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registration  and  provides  the  necessary  preliminary  documentation.  A  third  party  auditor 
conducts  a  registration  audit  and  a  series  of  follow-up  audits  to  ensure  that  identified 
nonconformities  have  been  adequately  addressed.  Once  a  supplier’s  nonconformities  have 
been  eliminated,  QS-9000  compliance  registration  is  conferred  on  the  supplier.  The 
validity  of  this  registration  is  maintained  by  surveillance  audits  conducted  every  6  months. 

C.  GOVERNMENT  PROGRAMS 

1.  Aimy — Contractor  Performance  Certification  Program 

The  Contractor  Performance  Certification  Program  (CP)2  represents  the  Army’s 
response  to  trends  within  the  Department  of  Defense  to  cut  government  and  contractor 
administrative  costs  while  increasing  levels  of  contractor  quality  and  performance.  (CP)2 
is  a  voluntary  supplier  recognition  program  in  which  the  Army  utilizes  an  established 
methodology  and  set  of  measures  to  appraise  contractors’  development,  production,  and 
maintenance  facilities  in  order  to  identify  those  contractors  with  a  commitment  to  quality, 
customer  satisfaction,  and  improved  products  and  processes.  The  relationship  that  the 
Army  builds  with  (CP)2  contractors  yields  the  following  benefits  for  the  Army  and  the 
government:  1)  the  means  to  ensure  that  acquisition  activities  involve  quality  products  that 
meet  the  needs  of  the  customer;  2)  greater  efficiency  and  decreasing  product  costs 
achieved  through  continual  improvements  in  product  and  process  quality;  and  3)  a 
reduction  in  the  need  for  government  oversight  and,  therefore,  declining  administrative 
costs. 

The  (CP)2  is  a  voluntary  program  open  to  any  contractor  that  has  pursued  or  made 
plans  to  pursue  contracts  with  the  Army  Materiel  Command’s  (AMC)  Major  Subordinate 
Commands  (MSCs).  The  contractor  commits  to  a  multiphase  assessment  process  that  may 
take  more  than  2  years  to  reach  a  conclusion  and  actually  bestow  (CP)2  status  upon  the 
contractor.  The  preliminary  phase  provides  the  contractor  with  information  on  (CP)2  and 
signals  the  contractor’s  formal  intent  to  enter  the  program.  During  the  preassessment 
phase,  the  contractor  submits  information  on  its  facilities,  organizational  structure,  and 
past  performance,  and  the  contractor  and  MSCs  determine  which  of  the  three  (CP)2 
“certification”  types  is  being  sought:  1)  production  certification;  2)  design  and 
development  certification;  and  3)  production,  design,  and  development  certification.  The 
assessment  phase  consists  of  a  baseline  and  final  assessment  by  an  assessment  team 
(composed  of  individuals  belonging  to  the  MSCs  and  contractor)  using  28  specifically 
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defined  criteria  and  a  series  of  additional  assessment  audits  requested  by  the  contractor  to 
verify  that  suggested  corrective  actions  have  been  implemented.  Recognition  happens 
only  after  all  of  the  MSCs’  concerns  have  been  adequately  addressed  and  a  memorandum 
of  agreement  is  issued  detailing  the  scope  and  responsibilities  of  the  contractor’s  (CP)2 
status. 

Once  the  contractor  has  received  (CP)2  status,  a  postassessment  phase  commences 
during  which  the  contractor  provides  periodic  reports  on  its  management  and  program, 
while  the  government  reserves  the  right  to  request  additional  complete  or  partial 
assessments  as  a  condition  for  extending  a  contractor’s  certification.  Despite  the 
extensive,  time-consuming  (CP)2  assessment  process,  contractors  seek  this  (CP)2  status 
because  they  benefit  from  the  resultant  reduced  scrap  and  rework,  decreased  customer 
complaints,  better  yield  rates,  improved  reputations  for  facilities,  increased 
competitiveness,  and  reduced  contract  administrative  costs. 

(CP)2  borrows  its  basic  framework  from  that  established  by  ISO  9000.  In  fact, 
(CP)2  uses  the  20  assessment  criteria  of  ISO  9000  as  part  of  its  own  assessment  criteria 
and,  further,  considers  them  a  minimum  level  that  must  be  achieved  with  respect  to 
(CP)2’s  first  20  criteria.  In  implementing  the  20  criteria  adopted  from  ISO  9000,  (CP)2 
places  additional  emphasis  on  enhancements  to  practices  and  principles.  Essentially, 
(CP)2  differs  from  ISO  9000  in  two  ways:  it  encourages  the  use  of  process  metrics  and 
places  greater  emphasis  on  collecting  and  documenting  physical  data.  In  addition  to  the 
20  criteria  adopted  from  ISO  9000,  (CP)2  uses  8  additional  criteria  to  assess  contractors: 
customer  satisfaction;  quality  costs;  warranty  performance;  ethics;  business  planning; 
safety;  enviromnental;  and  continuous  improvement  process. 

2.  FAA — Aircraft  Certification  Systems  Evaluation  Program 

The  Aircraft  Certification  Systems  Evaluation  Program  (ACSEP)  is  a  program  for 
the  comprehensive  evaluation  of  aviation  quality.  It  centers  on  auditing  aircraft  and 
aviation  industry  suppliers  based  on  Federal  Aviation  Regulations  requirements.  The 
Federal  Aviation  Administration  (FAA)  designed  the  ACSEP  both  to  ensure  that  its  own 
needs  and  the  needs  of  the  aircraft  industry  were  being  met,  and  to  adopt  a  system  of 
standard  methods  compatible  with  the  aircraft  manufacturing  and  internationally 
recognized  standards.  By  implementing  the  ACSEP,  the  FAA  has  established  a  set  of 
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standards  for  evaluating  quality  performance  and  a  means  of  collecting  information  on 
standards  applied  throughout  the  industry  over  and  above  those  presently  required  by  the 
FAA. 

Aircraft  industry  contractors  seeking  ACSEP  certification  are  assessed  by  a  team 
of  evaluators  who  have  completed  specialized  training  and  participated  in  a  number  of 
evaluations  as  trainees.  The  evaluations  consist  of  an  extensive  series  of  questions  that 
center  on  the  ACSEP’s  six  major  systems:  1)  quality;  2)  engineering;  3)  communications 
with  the  FAA;  4)  manufacturing;  5)  service  and  product;  and  6)  management.  Contractors 
are  further  assessed  on  the  basis  of  17  subsystems:  organization  and  responsibility;  design 
data  control;  software  quality  assurance;  manufacturing  processes;  special  manufacturing 
processes;  statistical  quality  control;  nonconforming  material;  material  handling  and 
storage;  nondestructive  inspection;  tool  and  gauge;  testing;  internal  audit;  global 
production;  supplier  control;  airworthiness  determination;  Federal  Aviation  Regulations 
reporting  requirements;  and  manufacturing  facility. 

After  assessing  a  supplier  in  terms  of  the  ACSEP  major  systems  and  subsystems, 
the  evaluation  team  holds  a  postevaluation  conference  with  the  supplier’s  management  to 
review  the  findings.  If  a  nonconformance  is  identified  under  ACSEP’s  major  systems  or 
subsystems,  it  is  presented  to  the  audited  supplier  and  the  principal  inspector  assumes 
responsibility  for  monitoring  the  supplier  to  ensure  that  formal  corrective  actions  have 
been  implemented.  The  audit  evaluation  data  is  then  entered  into  an  ACSEP  data  base. 

3.  Defense  Logistics  Agency  (DLA) — DSCC  and  DISC 

a.  Defense  Supply  Center  Columbus — Qualified  Manufacturer’s  List 

The  Qualified  Manufacturer’s  List  (QML)  and  related  Qualified  Products  List 
(QPL)  were  established  to  help  the  government  identify  contractors  with  a  history  of 
demonstrated  quality.  But  unlike  the  other  quality  assessment  programs  described  so  far 
in  this  paper,  the  QML  qualifies  contractor  processes  and  materials — ^not  the  contractors 
themselves.  It  is  described  here  for  completeness. 

A  QML  is  established  and  compiled  in  relation  to  the  announcement  of  a  new 
specification  and  its  qualification  requirements.  A  specification  has  qualification 
requirements  when  a  contractor’s  products  and  processes  must  meet  certain  standards 
before  a  contract  is  awarded.  When  qualification  requirements  are  attached  to  a 
specification,  it  is  the  government  agency’s  responsibility  to  encourage  manufacturers  to 
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specification  and  its  qualification  requirements  are  announced,  it  is  the  manufacturer’s 
responsibility  to  request  a  qualification. 

Manufacturers  undergo  the  QML  process  to  test  their  processes  and  materials 
against  a  specification’s  requirements  because  future  acquisition  awards  will  be  granted 
only  to  those  contractors  with  processes  and  materials  that  have  been  tested  by  and 
included  in  the  QML.  The  formal  assessment  phase  of  the  QML  qualification  process 
involves  an  audit  of  the  manufacturer’s  facilities  and  a  series  of  tests  as  presented  in  the 
specification’s  qualification  requirements.  The  manufacturer  is  notified  of  the  testing 
results  and,  if  it  has  failed,  provided  with  an  explanation  as  to  why  the  testing  and  test 
results  were  not  approved.  A  manufacturer  whose  processes  and  materials  have  been 
determined  to  fulfill  the  specification’s  qualification  requirements  is  notified  in  writing 
that  the  particular  process  or  material  that  was  tested  will  appear  on  the  QML  for  the 
specification. 

QMLs  are  monitored  by  the  DLA’s  Defense  Supply  Center  Columbus,  Sourcing 
and  Qualifications  Unit  (DSCC-VQ).  Every  2  years,  the  DSCC-VQ  reviews  the 
specification  and  its  qualification  requirements  to  determine  whether  they  should  be 
continued  and  maintained.  Once  this  review  is  complete  and  a  specification  and  its 
qualification  requirements  have  been  retained,  the  manufacturer  whose  process  or  material 
appears  on  the  QML  must  affirm  or  reaffirm  its  status.  Depending  upon  the  specifics  laid 
down  by  the  qualification  requirements  of  a  specification,  a  manufacturer  will  have  to  do 
one  of  three  things:  1)  certify  its  qualification  status;  2)  submit  to  periodic  retesting;  or  3) 
undertake  a  complete  series  of  tests  to  requalify. 

b.  Defense  Industrial  Supply  Center — Qualified  Suppliers  List  for 
Distributors/Qualified  Suppliers  List  for  Manufacturers 

Distributors  and  manufacturers  can  pre-qualify  themselves  for  consideration  in 
future  acquisition  competitions  through  the  Qualified  Suppliers  List  (QSL)  program 
managed  by  the  Defense  Industrial  Supply  Center  (DISC).  This  program  evaluates 
distributors  and  manufacturers  for  specific  conunodity  groups  and  produces  two  types  of 
lists:  the  Qualified  Suppliers  List  for  Distributors  (QSLD)  and  the  Qualified  Suppliers  List 
for  Manufacturers  (QSLM).  These  lists  represent  distributors  and  manufacturers  that  have 
been  determined  to  possess  the  necessary  quality  and  control  practices  to  allow  source 
inspections  of  individual  contracts  and  pre-award  surveillance  to  be  replaced  by 
acceptance  of  these  distributors’  and  manufacturers’  commercial  business  practices.  This 
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acceptance  of  these  distributors’  and  manufacturers’  commercial  business  practices.  This 
is  a  similar  practice  to  industry’s  preferred  supplier  practices  or  supplier  certification 
programs  described  in  Section  I.D. 

The  resulting  benefits  of  the  QSLD  and  QSLM  are  improved  quality  and  decreased 
lead  times  for  product  delivery.  Furthermore,  the  QSLD  and  QSLM  are  expected  to  result 
in  declining  life  cycle  costs  and  increased  customer  satisfaction.  At  present,  however, 
there  are  only  three  commodity  group  areas  for  which  QSLDs  and  QSLMs  exist.  These 
commodity  areas  are  as  follows; 

•  Bulk  Metals  QSLD 

•  Class  3  Threaded  Fastener  National  Stock  Number  (NSN)  QSLD  and  QSLM 

•  Blind  Aerospace  Rivets  (Federal  Supply  Certification  [FSC]  5320)  QSLD  and 
QSLM 

A  QSLD  and  QSLM  are  expected  in  the  near  future  for  the  Fiber  Rope,  Twine,  and 
Tape  (FSC  4020)  commodity  group.  Furthermore,  consideration  is  being  given  to  the 
applicability  of  the  QSLD  or  QSLM  programs  to  the  O-Rings  (FSC  5330)  and  Builders 
Hardware  (Part  of  FSC  5340)  commodity  groups. 

To  be  considered  for  inclusion  on  a  QSLD  or  QSLM,  a  distributor  or  manufacturer 

must: 

•  Possess  and  use  a  documented  quality  program  that  meets  DISC  criteria 

•  Maintain  a  single  quality  control  program 

•  Have  a  Commercial  and  Government  Entity  code  (CAGE) 

•  Submit  a  re-qualification  application  within  120  days  of  qualification 
expiration 

DISC  assesses  distributors  and  manufacturers  on  the  basis  of  criteria  specifically 
established  for  a  commodity  group  in  order  to  determine  the  existence  of  process  controls. 
If  such  controls  exist  and  are  used  on  a  daily  basis,  and  if  the  distributor  and  manufacturer 
meet  all  of  the  DISC  criteria,  there  is  a  level  of  assurance  that  the  products  procured  will 
meet  specification  requirements.  With  the  criteria  having  been  met,  the  distributor  or 
manufacturer  is  placed  on  the  QSLD  or  QSLM  for  that  specific  commodity  group  and, 
therefore,  may  compete  with  other  QSLD  or  QSLM  members  for  future  DISC 
procurement  contracts  under  this  program’s  solicitations. 
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D.  QUALITY  SYSTEM  ASSESSMENT  PROGRAM  COMPARISONS 

Appendix  A  contains  lists  of  the  process,  auditor,  and  results  criteria  for  the 
programs  described  in  this  chapter.  The  tables  in  this  section  compare  those  criteria. 

1.  Process  Criteria 

Table  II-l  compares  the  process  criteria  of  the  eight  programs  described  in  this 
chapter.  Process  criteria  are  the  attributes  of  the  assessment  process  itself.  ISO  9000 
process  criteria  are  used  as  the  baseline. 

2.  Auditor  Criteria 

We  discovered  little  information  on  the  scope  and  breadth  of  auditor  criteria  during 
our  evaluation  of  quality  system  assessment  programs.  Most  of  the  programs  do, 
however,  call  for  registration  to  be  conferred  by  a  third  party.  When  a  quality  system 
assessment  program  allows  or  calls  for  a  third  party  registrar,  an  individual  auditor — 
independent  of  a  contracting  activity’s  customer  or  supplier — confirms  that  the  supplier 
has  met  or  exceeded  the  established  requirements  for  a  quality  system.  To  ensure  that  the 
third  party  auditor  is  capable  of  performing  quality  system  audits,  a  supplier  generally  will 
seek  an  auditor  whose  quality  system  auditing  abilities  have  been  approved  to  be  of  the 
highest  level.  Auditors  are  approved  in  two  ways — certification  and  accreditation — by 
two  different  organizations — ^The  American  Society  for  Quality  (ASQ)  and  the  Registrar 
Accreditation  Board  (RAB). 

The  ASQ  established  its  auditor  certification  program  in  1966  and  currently  runs 
several  programs  that  certify  auditors  under  the  following  categories:  quality  engineer, 
quality  auditor,  reliability  engineer,  quality  technician,  mechanical  inspector,  quality 
manager,  and  software  quality  engineer.  Auditor  certification  indicates  that  an  auditor  has 
quality  assurance  experience  and  has  been  tested  and  proven  to  possess  a  core  of 
knowledge  in  the  quality  system  auditing  concepts  and  a  specific  standard,  such  as  ISO 
9000. 
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LEGEND;  Shaded  boxes  represent  similarity  to  ISO  9000  definition  at  top  of  table. 
Text  explains  deviations  from  the  ISO  9000  definition. 


The  Registrar  Accreditation  Board  (RAB),  established  by  the  ASQ  in  1989  as  an 
independent  auditor  accreditation  body,  conducts  U.S.  third  party  auditor  accreditation  and 
certification.  Under  the  RAB  program,  auditor  accreditation  refers  to  approval  of  the 
written  policies  and  procedures  that  an  auditor  follows  to  perform  quality  audits.'^ 

The  criteria  followed  by  RAB  for  accrediting  auditors  incorporates  criteria 
requirements  used  by  the  European  Union,  European  Free  Trade  Association,  and  other 
world  accreditation  bodies.^  Auditors  seeking  accreditation  complete  a  RAB  application 
and  submit  to  an  initial  review  to  determine  if  they  meet  the  basic  criteria.  A  RAB 
auditing  team  investigates  the  auditor’s  office  and  evaluates  the  auditor’s  performance  of  a 
supplier  audit.  The  team  evaluates  any  resulting  corrective  actions  and  produces  a  report 
that  recommends  whether  to  grant  or  withhold  the  auditor’s  accreditation.  The  RAB 
Accreditation  Council  reviews  the  auditor’s  report  and  has  the  final  authority  over  the 
auditor’s  accreditation  status.  If  determined  to  have  successfully  fulfilled  the  relevant 
criteria  for  auditor  accreditation,  the  auditor  is  issued  a  certificate.  Auditors  are  required 
to  maintain  accreditation  by  continuing  to  participate  in  quality  audits  and  reapplying  for 
their  auditor  accreditation  status  on  an  annual  basis. 

The  RAB’s  program  to  certify  auditors  to  perform  quality  audits  is  based  on 
standards’  criteria,  such  as  those  in  the  ISO  9000  family.  Auditors  must  fulfill  basic 
requirements  in  the  categories  of  education,  training,  and  experience  in  the  field  to  be 
audited  and  quality  system  auditing.  An  auditor’s  qualification  level  determines  the  level 
to  which  the  auditor  may  be  certified.  RAB  certifies  auditors  at  three  levels:  1)  Quality 
Systems  Provisional  Auditor;  2)  Quality  Systems  Auditor;  or  3)  Quality  Systems  Lead 
Auditor.  This  three-level  system  allows  an  auditor  to  advance  as  quality  audit  knowledge 
and  experience  is  gained;  however,  an  auditor  is  allowed  to  apply  for  certification  at 
whichever  level  best  meets  his  or  her  present  qualifications. 

The  Quality  Systems  Provisional  Auditor  category  is  intended  for  those  individuals 
with  little  or  no  experience  participating  in  the  quality  audit  process.  The  individuals 


^  RAB  has  also  worked  to  enhance  the  U.S.  registrar  accreditation  system.  In  1991,  it  worked  with  the 
American  National  Standards  Institute  (ANSI)  to  establish  the  American  National  Accreditation 
Program  for  Registrars  of  Quality  Systems.  RAB  maintains  an  active  role  in  administrating  this 
program. 

^  Specifically,  RAB  incorporates  accreditation  criteria  from  the  following:  1)  ISO/IEC  Guide  48, 
Guidelines  for  Third-Party  Assessment  and  Registration  of  a  Supplier’s  Quality  System;  2)  EN  45012, 
General  Criteria  for  Certification  Bodies  Operating  Quality  Systems  Certification;  and  3)  ISO  1(X)11, 
Guidelines  for  Auditing  Quality  Systems. 
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must,  however,  have  completed  training  and  an  examination  on  the  fundamentals  of 
auditing  and  ISO  9000  or  other  quality  system  standard,  and  they  must  have  a  bachelor’s 
degree  or  lesser  degree  supplemented  by  relevant  work  experience.  Two  sponsors  are  also 
needed  to  attest  to  the  individual’s  personal  attributes.  An  individual  may  remain  a 
Provisional  Auditor  for  3  years,  at  the  end  of  which  the  requirements  to  become  an  Quality 
Systems  Auditor  must  have  been  satisfied;  otherwise  the  certification  status  will  be 
terminated. 

Aside  firom  advancing  from  Provisional  Auditor  status,  an  individual  may  apply 
directly  to  that  of  a  Quality  Systems  Auditor  level.  The  Quality  Systems  Auditor  is 
intended  for  those  individuals  who  have  already  amassed  some  years  of  auditing 
experience.  An  individual  applying  for  this  auditor  level  will  have  proven  his  or  her 
competence  in  the  fundamentals  of  auditing  and  ISO  9000  or  other  quality  system 
standard.  In  addition,  depending  upon  the  formal  education  level,  the  applicant  must  have 
already  observed  and  participated  in  a  prescribed  number  of  quality  audits.  These  audits 
may  or  may  not  have  been  witnessed  by  an  existing  Lead  Auditor,  but  if  one  wishes  to 
eventually  advance  to  the  Lead  Auditor  level,  it  is  advisable  that  the  audits  be  observed. 

Finally,  to  become  a  Quality  System  Lead  Auditor,  an  individual  must  have 
demonstrated  an  ability  to  both  participate  in  and  lead  quality  audits.  The  auditor  applicant 
must  have  completed  a  total  of  five  quality  audits  as  the  lead  auditor  on  a  team  of  at  least 
two  other  auditors.  Lead  auditor  status  is  conferred  upon  those  individuals  who  have 
shown  their  proficiency  with  such  quality  system  issues  as  documentation  review,  on-site 
auditing  activities,  and  producing  auditing  reports. 

3.  Results  Criteria 

Table  11-2  compares  the  results  criteria  of  the  programs  with  the  20  elements  of 
ISO  9000.  Results  criteria  are  those  characteristics  against  which  the  quality  system  is 
judged.  The  FAA  ACSEP,  DSCC  QML/QPL,  and  DISC  QSLM/QSLD  programs  are  not 
included  in  the  table  because  their  results  criteria  are  too  different  from  the  other  programs 
for  their  inclusion  in  a  comparison  table  to  be  meaningful.  Details  about  the  last  column 
of  the  table.  Additional  Criteria  Beyond  ISO  20  Elements,  can  be  found  in  Appendix  A. 
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Table  11*2.  A  Comparison  of  Quality  Syatem  Assessment  Program  Criteria:  Results  Criteria  (page  1  of  7) 
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Table  11-2.  A  Comparison  of  Quality  System  Assessment  Program  Criteria:  Results  Criteria  (page  5  of  7) 
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Text  explains  deviations  from  the  ISO  9000  definition. 
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Table  11-2.  A  Comparison  of  Quality  System  Assessment  Program  Criteria:  Results  Criteria  (page  7  of  7) 
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LEGEND:  Shaded  boxes  represent  similarity  to  ISO  9000  definition  at  top  of  table. 
Text  explains  deviations  from  the  ISO  9000  definition. 


III.  QUALITY  AWARD  PROGRAMS 


Quality  awards  differ  from  the  programs  described  in  Chapter  n  because  they  are 
not  specific  to  the  contractor’s  quality  system — ^they  consider  the  quality  of  the  whole 
organization.  For  these  awards,  we  use  the  1997  Malcolm  Baldrige  National  Quality 
Award  as  the  baseline.  The  program  descriptions  include  the  three  criteria  areas — 
process,  auditor,  and  results — as  in  the  last  chapter.  A  comparative  section  is  included  at 
the  end.  All  summary  descriptions  are  taken  from  complete  texts  found  in  the  references 
listed  in  Appendix  D. 

A.  MALCOLM  BALDRIGE  NATIONAL  QUALITY  AWARD 

The  Malcolm  Baldrige  National  Quality  Award  (Baldrige  Award)  was  established 
in  1987  to  be  presented  annually  to  U.S.  companies  that  have  demonstrated  a 
commitment  to  quality  and  competitiveness.  Its  defining  objective  includes  fostering  a 
conunitment  to  quality,  awareness  of  its  importance  to  a  company’s  health  or  our  nation’s 
economy,  and  open  dialogue  among  companies  based  upon  their  knowledge  and 
experience  with  various  quality  strategies.  The  award  is  based  on  a  set  of  10  core  values: 
1)  customer-driven  quality;  2)  leadership;  3)  continuous  improvement  and  learning;  4) 
employee  participation  and  development;  5)  fast  response;  6)  quality  and  prevention  in 
design;  7)  long-range  view  of  the  future;  8)  management  by  fact;  9)  company 
responsibility  and  citizenship;  and  10)  results  focus. 

The  Baldrige  Award  represents  a  public-private  partnership  whereby  private 
sector  businesses  with  success  in  quality  management  are  evaluated  and  enlightened  by 
this  federally  funded  program  managed  by  the  National  Institute  of  Standards  and 
Technology  (NIST)  with  assistance  from  the  American  Society  for  Quality  (ASQ). 
Applicants  undergo  an  extensive  four-part  selection  process  that  includes  multiple, 
detailed  audits;  team  site  visit  evaluations;  and  a  final  panel  review  to  determine  how 
they  rate  on  the  20  award  criteria  across  seven  categories:  leadership;  strategic  planning; 
customer  and  market  focus;  information  and  analysis;  human  resource  development  and 
management;  process  management;  and  business  results. 
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Whereas  the  current  ISO  9000  is  a  system  of  quality  standards  for  the  purpose  of 
establishing  conformance,  the  Baldrige  Award  seeks  to  promote  successful  quality 
strategies  that  are  thought  of  as  essential  to  maintaining  and  increasing  the 
competitiveness  of  American  companies  in  the  global  marketplace.  The  Award  and  its 
criteria  are  intentionally  designed  to  be  educational  rather  than  prescriptive,  so  as  to 
encourage  innovation.  As  a  result,  many  companies  not  only  seek  to  win  this  award,  but 
also  actually  use  the  Baldrige  Award  criteria  and  core  values  to  evaluate  where  they  stand 
with  respect  to  other  businesses  and  assist  in  developing  and  implementing  new  business 
strategies.  Those  who  have  studied  and  learned  from  the  Baldrige  Award  program  have 
cited  increased  productivity,  a  larger  market  share,  and  greater  customer  satisfaction  as 
just  a  few  of  the  benefits  accrued  from  their  participation. 

B.  DEMING  APPLICATION  PRIZE  OF  THE  JAPANESE  UNION  OF 
JAPANESE  SCIENTISTS  AND  ENGINEERS 

The  Deming  Prize  was  created  in  1951  by  the  Japanese  Union  of  Scientists  and 
Engineers  (JUSE)  in  honor  of  Dr.  W.  E.  Deming  for  his  efforts  in  introducing  the 
concepts  of  quality  control,  specifically  statistical  quality  control,  to  Japan.  Statistical 
quality  control  was  presented  to  Japan  during  the  post-World  War  n  era  as  an  essential 
means  for  Japanese  economic  reconstruction  through  the  improvement  of  product  quality 
and  productivity  and  reduction  of  production  costs.  The  Deming  Prize,  which  actually 
refers  to  a  series  of  three  different  prizes  (the  Deming  Prize,  Deming  Application  Prize, 
and  Deming  Application  Award  for  Factory),  recognizes  an  individual,  a 
business/division,  or  a  factory  (depending  on  the  specific  prize)  that  has  researched  and 
disseminated  knowledge  on  the  subject  of  quality  control  or  implemented  successful 
quality  control  strategies. 

C.  PRESIDENT’S  AWARD  FOR  QUALITY  AND  PRODUCTIVITY 
IMPROVEMENT 

The  Presidential  Award  for  Quality  (President’s  Award)  was  conceived  in  1988  as 
a  means  of  identifying  and  recognizing  government  organizations  within  the  executive 
branch  for  successes  and  improvements  in  performance.  The  President’s  Award 
recognizes  federal  organizations  for  improving  performance  and  providing  high  quality 
services  through  the  efficient  use  of  taxpayer  dollars,  while  also  promoting  awareness  and 
encouraging  the  sharing  of  best  practices  with  regard  to  government  management 
techniques. 
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The  Office  of  Personnel  Management  (0PM)  administers  the  program  that 
annually  bestows  the  President’s  Award  and  its  related  Quality  Improvement  Prototype 
Award.  Applicants  for  the  President’s  Award  must  be  divisions  or  agencies  of  the 
executive  branch  and  have  at  least  100  employees.  The  evaluation  process  involves  a 
preliminary  eligibility  review  by  the  0PM,  a  team  evaluation,  a  group  site  visit 
inspection,  and  a  final  review  by  a  panel  of  judges  which  then  makes  reconunendations  to 
the  President  for  a  final  decision  on  the  award  recipients.  Throughout  the  review, 
applicants  are  evaluated  based  on  24  award  criteria  across  7  categories:  leadership; 
information  and  analysis;  strategic  planning;  human  resource  development  and 
management;  process  management;  business  results;  and  customer  focus  and  satisfaction. 

The  President’s  Award,  like  the  Baldrige  Award,  represents  a  nonprescriptive, 
educational  approach  to  the  promotion  of  quality  and  performance  and  is  not  a  scheme  for 
quality  management  system  standard  conformance,  such  as  ISO  9000.  This  educational 
focus  is  reflected  in  the  award  criteria,  which  are  based  on  11  key  values:  customer- 
driven  quality;  leadership;  continuous  improvement  and  learning;  employee  participation 
and  development;  fast  response;  design  quality  and  prevention;  long-range  view  of  the 
future;  management  by  fact;  partnership  development;  corporate  responsibility  and 
citizenship;  and  results  orientation.  It  is  not  coincidence  that  the  approach,  award  criteria, 
and  underlying  values  of  the  President’s  Award  parallel  those  of  the  Baldrige  Award. 
The  former  is  modeled  as  a  federal  government  adaptation  of  the  Baldrige  Award’s 
commitment  to  competitiveness  and  quality  in  the  private  sector. 

D.  GEORGE  M.  LOW  AWARD:  NASA’S  QUALITY  AND  EXCELLENCE 
AWARD 

The  George  M.  Low  Award  (GML  Award)  is  bestowed  by  the  National 
Aeronautics  and  Space  Administration  (NASA)  as  the  highest  honor  in  the  aerospace 
industry  in  recognition  of  quality,  productivity,  and  performance.  NASA  believes  that  a 
successful  U.S.  aerospace  program  depends  upon  NASA’s  access  to  domestically 
produced  products  and  services  of  high  quality.  The  GML  Award’s  three  overriding 
goals  are  to  1)  generate  awareness  of  quality  issues  and  their  importance;  2)  encourage 
U.S.  businesses  to  adopt  strategies  that  improve  company  quality  and  productivity,  as 
well  as  U.S.  competitiveness;  and  3)  create  an  arena  in  which  successful  quality  strategies 
and  techniques  may  be  shared. 
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The  GML  Award  may  be  pursued  by  any  prime  contractor  or  subcontractor  with 
NASA  that  meets  an  established  sales  figure,  number  of  employees,  and  other  specific 
requirement  criteria  with  respect  to  that  portion  of  their  business  that  relates  directly  to 
their  work  with  and  for  NASA.  Of  those  eligible,  candidates  are  nominated  annually  to 
compete  for  a  total  of  up  to  four  awards  representing  two  classifications  (large  and  small 
businesses)  and  two  categories  (products  and  services).  Once  the  preliminary 
nominations  have  been  made,  the  NASA  Strategic  Enterprise  GML  Review  Council 
determines  10  finalists.  These  companies  then  submit  additional  documentation  with 
respect  to  nine  award  criteria  over  seven  categories:  1)  NASA  contract  performance  and 
customer  satisfaction,  2)  NASA  schedule,  3)  NASA  cost,  4)  long-term  organizational 
initiatives  to  respond  to  NASA’s  strategic  aspirations,  5)  leadership  and  continuous 
improvement,  6)  innovative  management  and  technology  breakthroughs,  and  7)  items  of 
special  interest  to  NASA.  Site  visits  are  conducted  and  recommendations  made  to  a 
GML  panel  of  judges  that  selects  the  winners  who  are  ultimately  approved  by  the  NASA 
Administrator. 

Similar  to  other  quality  awards,  the  GML  Award  is  not  a  scheme  of  quality 
management  system  conformance  standards  like  ISO  90(X).  Instead,  the  award 
encourages  knowledge  sharing  within  the  aerospace  industry  and  strives  to  recognize 
those  companies  that  have  adopted  strategies  that  foster  quality  and  successes  in  NASA’s 
programs.  In  general,  the  GML  Award’s  criteria  categories  are  much  more  broadly 
defined  than  those  of  the  Baldrige  and  President’s  Awards,  yet  they  are  much  more 
specifically  focused  since  they  evaluate  only  those  companies  involved  in  NASA 
contracts. 

E.  SHIGEO  SHINGO  PRIZE  FOR  EXCELLENCE  IN  MANUFACTURING 

In  1988,  the  Shingo  Prize  for  Excellence  in  Manufacturing  (Shingo  Prize)  was 
established  to  recognize  businesses  that  adhere  to  those  ideals  held  by  the  prize’s 
namesake,  Shigeo  Shingo,  a  leading  expert  on  quality  and  improving  the  manufacturing 
process.  The  award’s  philosophy  is  that  a  business  or  manufacturer  can  compete 
successfully  in  the  world  market  only  if  it  systematically  works  toward  continuous 
improvement  of  the  core  manufacturing  process  through  such  methods  as  lean 
manufacturing,  just-in-time  systems,  waste  reduction,  and  defect  control.  Accordingly, 
the  Shingo  Prize  objectives  are  to  1)  provide  a  forum  for  generating  awareness  of  various 
manufacturing  methods  and  systems  in  order  to  increase  competitiveness;  2)  encourage 


the  sharing  of  successful  manufacturing  strategies  and  methods;  and  3)  promote  research 
on  manufacturing  processes  and  production  among  business  leaders  and  in  academia. 

Administered  by  a  partnership  between  the  College  of  Business  at  Utah  State 
University  and  the  National  Association  of  Manufacturers  (NAM),  the  Shingo  Prize 
selects  winners  annually  from  a  nomination  pool  of  large  and  small  manufacturers  from 
the  United  States,  Canada,  and  Mexico.  The  evaluation  process  includes  the  individual 
and  joint  reviews  of  nominees’  “achievement  reports”  by  several  members  of  the  Board 
of  Examiners,  site  visits  by  five  to  six  examiners,  and  the  final  review  and  ratification  by 
the  Shingo  Prize  Council.  The  manufacturers  are  evaluated  on  the  basis  of 
strategy/implementation  and  results  dimensions  with  regard  to  10  key  criteria  across  4 
categories:  total  quality  and  productivity  management  culture  and  infrastructure; 
manufacturing  strategy,  processes,  and  systems;  measured  quality  and  productivity;  and 
measured  customer  service.  In  addition  to  rewarding  manufacturers  for  quality  successes, 
the  Shingo  Prize  program  furthers  its  educational/knowledge  sharing  goals  by  awarding  a 
Shingo  Prize  for  research  and  professional  publication  in  the  areas  of  manufacturing 
quality,  productivity,  and  process  improvement. 

F.  AWARDS  PROGRAM  COMPARISONS 

1.  Process  Criteria 

Table  m-l  compares  the  process  criteria  of  the  five  quality  awards  programs 
described  in  this  chapter  with  the  Malcolm  Baldrige  criteria,  the  baseline. 

2.  Auditor  Criteria 

Of  the  quality  award’s  reviewed  for  this  report,  the  Baldrige  Award  was  the  only 
program  to  explicitly  outline  the  details  of  the  process  required  to  become  an  auditor.  In 
general,  Baldrige  Award  auditor  applicants  are  expected  to  possess  formal  educational 
and  practical  experience  related  to  quality  and  methods  for  continuous  improvements. 
Auditors  are  evaluated  for  the  Baldrige  program  based  on  several  factors:  1)  their  depth 
of  experience  with  the  issues  and  concepts  represented  by  the  seven  results  criteria 
categories  of  the  Baldrige  Award;  2)  their  diversity  of  experience  as  shown  in  experience 
across  different  economic  sectors  and/or  industries;  3)  a  demonstration  of  leadership 
qualities  and  the  ability  to  communicate  effectively  in  written  form  or  verbally; 
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Table  III-1.  A  Comparison  of  Quality  Awards’  Criteria:  Process  Criteria  (page  1  of  2) 
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Table  IIM.  A  Comparison  of  Quality  Awards’  Criteria:  Process  Criteria  (page  2  of  2) 
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4)  knowledge  of  quality  concepts  and  their  relation  to  processes  and  results;  and  5)  the 
two  references  provided.  Applicants  selected  for  consideration  must  attend  a  3-day 
course  that  familiarizes  them  with  the  specific  criteria,  evaluation  system,  site  visit 
procedures,  etc.,  of  the  Baldrige  Award.  Furthermore,  the  applicants  will  undergo  a  20- 
to  40-hour  evaluation  of  their  performance  during  a  quality  case  study.  The  evaluation 
process  for  identifying  Baldrige  Award  auditors  takes  place  on  an  annual  basis,  with 
roughly  a  third  of  the  auditors  being  replaced  each  year  in  order  that  the  auditor  pool  may 
reflect  the  diversity  of  business  size  and  sectors. 

The  Deming  Prize  takes  a  much  less  formal  assessment  approach  in  identifying  its 
auditor  pool  since  the  Deming  Prize  committee  members  are  appointed  to  their  unpaid 
positions  by  the  committee’s  chair,  who  is  also  the  President  of  the  Japanese  Union  of 
Scientists  and  Engineers.  The  committee  members  are  chosen  to  draw  individuals  of 
differing  backgrounds,  including  government,  media,  and  business. 

The  President’s  Award  is  closely  linked  to  the  Baldrige  Award  in  the  structure  of 
its  results  and  process  criteria,  but  very  little  is  detailed  on  the  process  required  to  become 
an  auditor  of  this  program.  The  selected  auditors  are,  however,  representative  of  both 
public  and  private  sector  organizations. 

The  GML  Award  assembles  its  pool  of  auditors  and  reviewers  from  existing 
representatives  of  the  strategic  enterprises  and  the  NASA  headquarters  office. 

Auditors  for  the  Shingo  Prize  are  representative  of  business,  government,  and 
academia.  These  individuals  possess  knowledge  of  manufacturing  and  its  related 
processes,  systems,  and  methods  for  improvement.  The  Shingo  Prize  program,  like 
Baldrige,  does  provide  formal  training  to  its  auditors. 

3.  Results  Criteria 

Table  in-2  compares  the  results  criteria  using  the  1997  Baldrige  Award  criteria  as 
the  baseline.  Numbers  in  the  table  refer  to  criteria  sections  in  the  award  or  prize 
document. 
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Table  III-2.  A  Comparison  of  Quality  Awards’  Criteria:  Results  Criteria  (page  1  of  7) 
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IV.  PAST  PERFORMANCE  PROGRAMS 


This  research  effort  initially  set  out  to  compare  programs  that  assess  contractors’ 
quality  systems  and  supplier  quality  in  general.  We  found,  however,  that  some  elements 
of  past  performance  programs  could  pertain  to  our  consideration  of  reciprocal  agreements 
and  decided  to  examine  past  performance  programs  as  well.  For  example,  the  Navy’s 
RedAfellow/Green  Program  has  a  reciprocal  agreement  with  the  Coordinating  Agency  for 
Supplier  Evaluations  (C.A.S.E.),  a  nonprofit  corporation  that  promotes  cost  savings 
through  the  reduction  of  redundant  supplier  audits  and  assessments.  C.A.S.E.  uses  an 
extensive  data  base  for  sharing  nonprejudicial  supplier  information,  and  data  bases  are 
important  factors  for  keeping  past  performance  information  as  well. 

A.  DoD  POLICY 

When  ordinary  customers  make  purchase  decisions  in  the  marketplace,  they 
consider  their  history  with  merchants  or  suppliers.  Similarly,  a  13  October  1994 
executive  order  signaled  that  the  government  would  increase  its  emphasis  on  the  past 
performance  of  contractors.  In  1996,  the  Department  of  Defense  (DoD)  announced  that  it 
would  look  beyond  cost  considerations  to  consider  such  additional  factors  as  price, 
quality,  delivery  performance,  and  service  in  considering  all  its  acquisition  activities  over 
a  certain  dollar  threshold.  With  that  mandate,  the  past  performance  of  a  contractor 
became  a  vital  criterion  in  the  determination  of  a  DoD  contract  award. 

According  to  a  17  June  1996  report  by  Arthur  D.  Little,  Inc.,  35  systems  existed 
across  DoD  and  its  components  dealing  with  past  performance  information  (PPI).  These 
PPI  systems  lacked  uniformity,  however,  especially  in  their  approach  to  the  collection 
and  use  of  PPI.  In  order  to  design  and  implement  a  more  uniform  PPI  approach  that 
would  address  the  specific  needs  of  DoD  as  a  whole  and  its  unique  business  areas,  an 
Integrated  Product  Team  (IPT)  was  established  in  February  1997.  This  Past  Performance 
BPT’s  focus  was  the  promotion  of  joint  endeavors  between  component  and  civilian 
agencies  and  the  use  of  electronic  data  bases  for  the  collection  and  dissemination  of  PPI 
data. 


IV-1 


The  IPT  has  since  defined  the  business  areas  encompassed  by  DoD  and  the  past 
performance  factors  associated  with  each  of  these  areas.  This  approach  mirrors  that 
expounded  in  the  Little  report,  which,  because  of  the  sheer  size  and  diversity  of  the 
defense  business,  argued  that  DoD-related  business  areas  should  be  defined  along  with 
their  corresponding  strategies  and  evaluations.  By  approaching  the  simplification  and 
improvement  of  DoD’s  PPI  collection  and  use  at  the  business  area  rather  than 
departmental  level,  the  IPT  ensured  that  a  business  area  would  have  access  to  the 
contractor  past  performance  data  necessary  to  make  informed  contract  awards. 

In  a  20  November  1997  memorandum,  the  Under  Secretary  of  Defense  for 
Acquisition  and  Technology,  Dr.  Jacques  Gansler,  announced  the  creation  of  a  policy 
refinement  to  the  Federal  Acquisition  Regulation  (FAR)  Parts  15,  19, 42  and  the  Defense 
Federal  Acquisition  Regulation  Supplement  (DFARS)  Part  36.  This  policy  change, 
effective  1  February  1998,  states  that  DoD  shall  use  a  consistent  management  approach 
for  the  collection  and  use  of  PPI.  This  approach  includes  contract  dollar  thresholds 
tailored  by  business  sector,  an  established  set  of  contractor  assessment  elements,  and  an 
element  rating  system.  The  five-level  assessment  rating  system  applies  to  all  business 
sectors,  except  Construction  and  Architect-Engineering,  and  is  to  be  used  in  completing 
report  cards  for  all  PPI  assessment  elements.  Since  the  existence  of  an  automated  system 
for  the  collection  and  retrieval  of  PPI  is  key  to  the  success  of  this  policy’s 
implementation,  the  Life  Cycle  Information  Integration  Office  has  been  tasked  to  conduct 
a  pilot  effort  to  identify  the  current  DoD  PPI  interfaces  and  show  the  ability  to  integrate 
PPI  collection  efforts.  A  plan  for  the  automation  of  the  PPI  collection  and  retrieval 
process  within  DoD  is  to  be  presented  to  the  USD(A&T)  on  30  March  1998. 

In  the  meantime,  however,  several  existing  past  performance  systems  are  worthy 
of  further  investigation.  These  systems  include  the  Air  Force’s  Contractor  Performance 
Assessment  Reporting  System,  the  DoD-wide  Deficiency  Reporting  System,  the  Navy’s 
Product  Deficiency  Reporting  and  Evaluation  Program,  and  the  Navy’s 
Red/Yellow/Green  Program. 

B.  CONTRACTOR  PERFORMANCE  ASSESSMENT  REPORTING  SYSTEM 

The  Contractor  Performance  Assessment  Reporting  System  (CPARS)  is  the  Air 
Force  mechanism  for  appraising  contractors  based  on  their  past  performance.  The 
CPARS  is  a  data  base  containing  background  information  and  contractor  performance 
evaluations  on  an  ongoing  basis.  Through  this  system,  each  contractor’s  strengths  and 
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weaknesses  are  highlighted  by  specific  performance  measurement  data  and  project 
managers’  narrative  assessments  of  contractor  performance.  Furthermore,  each 
contractor  is  assigned  an  overall,  color-coded  performance  rating  (blue — exceptional; 
green — satisfactory;  yellow — ^marginal;  or  red — unsatisfactory)  within  CPARS. 

CPARS  is  used  in  major  Air  Force  acquisitions  of  more  than  $5  million. 
Government  personnel  involved  in  acquisition  activities  retrieve  information  from 
CPARS  when  evaluating  contractors  during  the  source  selection  process.  In  addition  to 
providing  background  on  the  contractor’s  program  and  efforts,  CPARS  gives,  for  each  of 
a  contractor’s  contracts,  the  contractor’s  evaluation  rating,  which  is  based  on  numerous 
measurement  elements;  product  and  system  performance,  cost  performance,  product 
assurance,  testing  and  evaluation,  the  responsiveness  of  management,  and  the 
management  of  subcontractors. 

The  CPARS  involves  the  generation  of  a  series  of  reports.  An  initial  report  is 
mandatory  for  new  contracts  and  provides  an  evaluation  of  a  contractor’s  performance 
during  at  least  the  first  180  days  of  a  contract.  Intermediate  reports  are  then  required  on 
an  annual  basis  throughout  the  duration  of  the  contract.  Occasionally,  an  out-of-cycle 
report  will  be  deemed  necessary  by  a  program  manager  in  order  for  the  contractor’s 
evaluation  to  reflect  any  significant  performance  changes  that  have  occurred  since  the 
previous  report.  At  the  completion  of  a  contract,  through  the  delivery  of  the  contracted 
item,  the  transfer  of  authority  for  a  program’s  management,  or  a  contract  termination,  a 
final  report  that  details  the  contractor’s  performance  since  the  most  recent  report  is 
released. 

Contractors  may  access  their  own  measurement  data  and  evaluation  information 
contained  in  CPARS.  Contractors  can  review  and  comment  on  their  evaluations  through 
CPARS.  They  may  also  use  the  CPARS  information  as  valuable  performance  feedback 
in  their  efforts  to  identify  areas  in  which  further  improvements  are  needed. 

C.  DEnCIENCY  REPORTING  SYSTEM 

The  Deficiency  Reporting  System  (DRS)  was  designed  as  a  DoD-wide  system  for 
reporting  complaints  and  corrective  actions  associated  with  contract  shortages,  overages, 
and  the  use  of  incorrect  material.  The  DRS  is  an  attempt  to  simplify  the  process  through 
which  action  officers  receive  information  regarding  a  contract’s  deficiencies  or 
discrepancies. 
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The  DRS  combines  data  received  across  DoD  through  the  use  of  several  existing 
reports,  such  as  Reports  of  Discrepancy  (ROD),  Product  Quality  Deficiency  Reports 
(PQDRs),  and  Transportation  Discrepancy  Reports  (TDRs).  The  DRS  is  also  integrated 
with  existing  systems  for  the  collection  of  specific  data  related  to  contractor  deficiencies 
and  complaints.  These  systems  include  the  Stock  Control  System  (SCS),  the 
Provisioning  and  Cataloging  Technical  Support  System  (PCTSS),  Depot  Maintenance 
Systems  (DMS),  and  Standard  Procurement  System  (SPS). 

When  a  complaint  is  lodged  by  a  customer  against  a  DoD  contractor,  this 
information  is  routed  into  the  DRS.  Once  received,  the  discrepancy  or  deficiency 
outlined  in  the  complaint  is  evaluated  and  a  corrective  action  is  devised.  Corrective 
actions  are  implemented,  any  discrepant  or  deficient  materiel  is  identified,  and  steps  are 
taken  to  ensure  its  appropriate  disposal.  Furthermore,  the  DRS  provides  an  opportunity 
to  measure  and  analyze  deficiency  and  discrepancy  trends  for  DoD  contractors. 

D.  PRODUCT  DEFICIENCY  REPORTING  AND  EVALUATION  PROGRAM 

The  Product  Deficiency  Reporting  and  Evaluation  Program  (PDREP)  was 
established  in  the  mid-1980s  as  the  Navy’s  centralized  system  for  data  on  contractors’ 
past  performance,  specifically  with  regard  to  product  quality.  The  PDREP  is  an 
automated  system  focused  on  the  quality  and  delivery  performance  of  the  Navy’s 
procurement  activities.  Components  of  the  PDREP  include  a  data  base  of  past 
performance  data,  profiles  on  contractor  performance,  and  a  system  for  evaluating 
contractors. 

Data  for  inclusion  in  the  PDREP  is  collected  at  the  materiel  level.  An  individual 
contractor’s  evaluation  is  then  based  on  the  aggregate  of  the  past  performance  data 
collected  on  that  contractor’s  products  for  each  applicable  materiel  category.  Purchasing 
officers,  inspectors,  and  quality  assurance  personnel  may  access  this  information  through 
standard  or  individualized  reports  to  aid  in  the  source  selection  process  of  contract 
awards. 

The  PDREP  gathers  its  contractor  data  through  a  diverse  array  of  means, 
including  Product  Quality  Deficiency  Reports  (PQDRs),  Material  Inspection  Records 
(MIR),  surveys,  testing  reports.  Contract  Delivery  Data  (CDD),  and  requests  for 
corrective  action.  This  data  is  collected  from  the  Systems  Commands  and  updated  on  a 
daily  basis. 
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E.  RED/YELLOW/GREEN  PROGRAM 


The  Red/Yellow/Green  Program  (RYG)  was  created  by  the  U.S.  Navy  as  a  means 
of  addressing  Section  9.104-3c  of  the  Department  of  Defense  FAR  Supplement,  which 
addressed  quality  as  an  important  element  when  reviewing  a  contractor’s  past 
performance  or  selecting  a  contractor  for  present  or  future  contracts.  The  RYG  is  a  series 
of  procedures  and  a  computerized  system  for  assigning  red,  yellow,  and  green 
classifications  to  a  contractor’s  Federal  Supply  Classifications  (FSC),  which  can  then  be 
utilized  in  the  source  selection  process.  The  importance  of  a  specific  contractor’s  quality 
history  is  therefore  emphasized,  as  the  RYG  provides  the  means  with  which  to  include 
the  cost  of  receiving  and  maintaining  poor  quality  goods  and  services  during  source 
selection. 

The  RYG  is  open  to  any  contractor  involved  in  materiel  procurement  contracts 
with  the  Navy.  The  program  does  not  classify  contractors,  but  rather  contractors’  quality 
performances  with  respect  to  individual  FSCs.  A  single  contractor  can  therefore  receive 
varying  classifications  at  any  one  time  depending  on  the  quality  status  of  its  FSCs.  The 
RYG  relies  upon  data  collected  from  contractors  and  maintained  in  the  Navy’s  Contractor 
Evaluation  System  (CES)  and  the  Product  Deficiency  Reporting  and  Evaluation  Program 
(PDREP).  Every  month,  the  RYG  accesses  these  data  bases  and  uses  the  data  to  classify 
each  contractor’s  FSCs  on  the  basis  of  risk  to  the  government  if  poor  quality  products  are 
received  (red — ^high  risk;  yellow — ^moderate  risk;  green — ^low  risk;  neutral — ^insufficient 
data).  On  a  monthly  basis,  the  contractors  are  then  apprised  of  their  RYG  classifications 
and  provided  the  opportunity  to  challenge  the  results. 

The  RYG  is  considered  a  source  selection  enhancement  program  since  it  allows 
the  Navy  to  review  and  assess  quality  data  that  can  then  be  used  in  the  procurement 
selection  process.  The  RYG  classifications  are  used  in  conjunction  with  two  equations 
that  assist  in  factoring  quality  issues  into  the  contract  award  process:  Technical 
Evaluation  Adjustment  (TEA)  and  Greatest  Value/Best  Buy  (GV/BB).  The  TEA  is 
actually  a  formula  that  allows  a  contracting  officer  to  determine  the  costs  to  government 
of  having  to  implement  additional  quality  assiu^ce  actions  associated  with  a  contractor’s 
RYG  classification  for  a  particular  FSC.  The  GV/BB  provides  yet  another  means  to  add 
RYG  classifications  to  price  as  evaluation  factors  in  the  contract  award  decision-making 
process. 
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V.  ESTABLISfflNG  RECIPROCAL  AGREEMENTS 


The  formation  of  reciprocal  agreements  to  eliminate  costly  redundant  audits  of 
quality  systems  is  facilitated  when  those  systems  are  similar  across  a  wide  range  of 
companies.  The  defense  industrial  base,  however,  has  a  range  of  systems  for  assuring 
quality.  A  previous  IDA  study  quoted  a  1989  Defense  Contract  Management  Coimnand 
(DCMC)  review  that  determined  that,  of  the  plants  in  which  DCMC  performed  in-plant 
quality  assurance  activities,  800  facilities  met  the  old  MIL-Q-9858,  Quality  System 
Requirements-,  7,200  facilities  met  MIL-I-45208,  Inspection  System  Requirements;  and 
8,000  facilities  worked  to  the  simple  standard  inspection  clause.*  Much  has  changed 
since  the  time  of  that  review.  MBL-I  and  MIL-Q  have  been  canceled,  and  DoD  policy 
under  Acquisition  Reform  and  the  Single  Process  Initiative  is  to  allow  each  contractor 
facility  to  use  the  single  quality  system  that  best  meets  its  customers’  needs.  Although 
the  current  trend  is  to  move  toward  process  controls  and  away  from  end-item  inspection 
in  managing  product  quality,  the  cost  of  changing  their  quality  system  is  an  issue  for 
many  contractors. 

A  May  1997  report  of  the  precision  gear  industry  published  the  findings  of  a  study 
that  discusses  this  cost  issue,  assesses  the  feasibility  of  a  Common  Quality  Certification 
System  (CQCS),  and  reconunends  how  to  proceed  in  developing  such  a  system.  We 
begin  by  sununarizing  that  report,  which  provides  some  specific  guidance  on  reciprocity 
in  auditing  and  certifying  suppliers.  We  then  describe  two  existing  reciprocal  agreement 
organizations  and  the  quality  system  assurance  process  for  NATO  countries.  We 
conclude  with  a  description  of  current  DoD  policy  and  a  recommendation  from  the 
Government  and  Industry  Quality  Liaison  panel  (GIQLP). 

A.  COMMON  QUALITY  CERTinCATION  SYSTEM  REPORT 

In  May  1997,  the  HT  Research  Instimte  published  a  smdy  report,  “Common 
Quality  Certification  System,”  in  which  it  assesses  the  effect  of  Defense  Acquisition 


*  Karen  J.  Richter  and  Seymour  J.  Lorber,  New  Quality  Assurance  Practices,  IDA  Paper  P-2991,  August 
1994. 
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Reform  on  the  aerospace  precision  gear  industry,  specifically  with  regard  to  quality  and 
specification  reform.  The  report,  which  was  done  for  the  U.  S.  Army  Aviation  and  Troop 
Command’s  Instrumented  Factory  for  Gears  (INF AC),  explores  six  basic  questions  and 
recommends  whether  and  how  to  proceed  in  developing  a  Common  Quality  Certification 
System  (CQCS). 

1.  CQCS  Study  Approach 

The  study  approach  included  research,  interviews,  and  analysis.  The  effort  began 
with  research  on  Defense  Acquisition  Reform,  the  Single  Process  Initiative,  and  other 
related  quality  and  specification  issues.  Interviews  were  then  conducted  with  seven 
companies  representing  all  levels  of  the  aerospace  precision  gear  industry  to  assess  their 
experience  with  quality  systems  and  certification.  The  results  of  the  research  and 
interviews  were  analyzed  to  answer  the  established  project  questions,  summarized  below. 

1)  What  impact  has  the  change  from  military  to  commercial  specifications  had 
on  manufacturing  processes?  The  conversion  from  military  to  commercial  specifications 
has  brought  about  a  parallel  shift  in  quality’s  emphasis  from  inspection  to  process 
control.  Although  inspection  does  remain  a  valuable  quality  system  component,  its 
emphasis  has  diminished.  Quality  systems,  in  general,  have  moved  from  their  traditional 
reactive  approach  to  become  more  proactive. 

2)  Are  the  commonly  identified  quality  cost  drivers  valid?  The  study  confirmed 
quality’s  major  cost  drivers  to  be  the  conduct  of  many  audits,  requiring  labor  and 
paperwork  expenditures;  oversight;  and  the  expense  of  time  and  labor  necessary  to 
remain  current  on  government  specifications  and  their  changes.  One  company  reportedly 
expended  over  $100,000  annually  and  as  many  as  77  days  to  conduct  just  the  external 
assessment  phase  of  certification.  Companies  seemed  to  indicate  that  a  CQCS  could 
alleviate  some  of  these  costs,  but  also  cautioned  that  there  would  be  little  in  the  way  of 
cost  savings,  especially  to  pass  along  to  the  customer,  at  least  in  the  short  term.  Quality 
system  certification  is  a  time-consuming  and  expensive  process,  which  includes  not  only 
the  external  cost  of  audits,  but  also  the  internal  costs  expended  in  preparing  for  the  audit. 
Indeed,  in  considering  internal  costs,  one  company  estimated  that  it  takes  anywhere  from 
1  to  2  years  of  work  by  an  engineer  or  manager  to  prepare  a  company  for  a  certification 
audit. 

3)  What  are  the  risks  in  replacing  military  specifications  and  standards  with 
alternative  quality  methods?  None  of  the  companies  interviewed  indicated  that  they 
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associated  any  risk  with  using  quality  system  methods  other  than  military  specifications. 
They  did,  however,  express  some  apprehension  on  two  grounds.  First,  ISO  9000  does  not 
contain  all  the  requirements  detailed  in  the  military  specifications;  thus,  suppliers  might 
sacrifice  some  aspects  of  quality  in  order  to  compete  for  lowest  bidder  status  in  the 
awarding  of  contracts.  Second,  ISO  9000  might  not  meet  all  the  requirements  for  a 
CQCS. 

4)  What  contrasts  and  comparisons  may  be  drawn  in  assessing  commercial  and 
DoD  aerospace  gear  quality  practices  and  certification  systems?  Most  of  the  companies 
have  implemented  a  single  quality  system.  These  systems  are  most  often  based  upon 
military  specifications  or  ISO  9000  and  are  enhanced  by  additional  quality  requirements 
to  address  the  needs  of  their  most  stringent  customers. 

5)  What  key  quality  characteristics  and  processes  are  involved?  The  companies 
cited  heat  treatment,  materials,  blueprint  requirements,  gage  calibration,  power  system 
design,  etc. 

6)  What  characteristics  are  necessary  to  develop  a  military  and  commercial 
quality  and  certification  system?  A  CQCS  should  approach  quality  from  a  process 
orientation,  but  should  also  integrate  product  inspection  requirements.  During  the 
definition  phase,  such  a  system  should  look  to  ISO  9000’s  20  criteria  and  its  8  Basic 
Quality  Principles  (customer-focused  organization;  leadership;  involvement  of  people; 
process  approach;  system  approach  to  management;  continual  improvement;  factual 
approach  to  decision  making;  and  mutually  beneficial  supplier  relationships). 

2.  CQCS  Study  Recommendations 

The  report  recommends  specific  approaches  to  developing  a  CQCS. 

1)  Initially  define  the  CQCS  based  on  ISO  9000’s  20  criteria  and  8  Basic 
Quality  Principles  to  create  a  quality  system  and  certification  that  emphasizes 
process  control  while  also  recognizing  the  importance  of  the  limited  but 
efficient  use  of  inspection. 

2)  Encourage  companies  to  aid  their  process  control  procedures  by  defining  and 
confirming  their  key  product  and  process  characteristics.  Because  ISO  9000 
will  likely  represent  the  basis  for  the  development  of  a  CQCS,  U.S. 
companies  should  seek  to  play  a  more  active  role  in  the  U.S.  technical 
advisory  group  to  the  ISO  technical  committee,  which  develops  and 
maintains  the  ISO  standards. 
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3)  Make  the  CQCS  applicable  at  all  supplier  levels.  Primes  therefore  need  to 
undertake  efforts  to  reduce  the  redundancies  and  foster  reciprocity  in  the 
auditing  and  certification  of  their  own  suppliers.  Also,  the  primes  need  to 
agree  on  criteria  and  encourage  reciprocity  in  the  audit  and  certification  of 
their  suppliers.  Otherwise,  a  CQCS  will  provide  inconsistent  benefits  to  the 
industry  as  a  whole. 

B.  EXISTING  INDUSTRY  RECIPROCAL  AGREEMENT  ORGANIZATIONS 

Two  national  organizations  promote  reciprocal  recognition  of  quality  audits 
among  their  members — the  Coordinating  Agency  for  Supplier  Evaluations,  or  C.A.S.E., 
and  the  National  Aerospace  and  Defense  Contractors  Accreditation  Program,  or 
NADCAP.  C.A.S.E.  members  have  access  to  all  data  from  audits  performed  by  another 
member.  NADCAP  members  use  the  Performance  Review  Institute  (PRI)  for  all  audits. 
In  both  organizations,  members  maintain  control  over  the  criteria  and  requirements  of  the 
audits  and  the  auditors. 

1.  Coordinating  Agency  for  Supplier  Evaluations 

C.A.S.E.  is  a  nonprofit,  mutual  benefit  corporation  whose  purpose  is  to  “promote 
the  improvement  of  quality  and  the  reduction  of  costs  in  industry  for  the  benefit  of  major 
government  and  industrial  contractors  and  their  respective  customers  and  sources  of 
supplies  and  services.”  It  is  a  coalition  of  industrial  companies  dedicated  to — 

•  Exchanging  and  publishing  non-prejudicial  supplier  data 

•  Reducing  redundant  supplier  audits  or  assessments 

•  Standardizing  supplier  and  procurement  quality  practices 

•  Reducing  supplier  management  costs  through  expense  avoidance 

a.  Organization  Description 

C.A.S.E.  is  governed  by  a  Statement  of  Principles  and  Bylaws.  C.A.S.E. 
management  consists  of  a  voluntary  Board  of  Directors  with  elected  and  appointed 
officers.  The  president  and  vice  president  are  from  the  Board  and  elected  by  it.  The 
Board  appoints  the  executive  director,  secretary,  and  treasurer, 

C,A.S.E.  has  two  types  of  members — sustaining  and  associate.  Sustaining 
members  have  voting  rights  and  the  rights  to  receive  data  about  the  evaluated  sources  of 
supply  published  in  the  C.A.S.E.  data  base.  Associate  members  can  also  receive  the 
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C.A.S.E.  data  base  but  have  no  voting  rights.  A  sustaining  member,  upon  reviewing  the 
data  base  about  a  prospective  supplier,  can  submit  a  request  for  data  directly  from  the 
sustaining  member  who  performed  the  audit.  C.A.S.E.  coordinates  the  request. 

The  organization  is  divided  into  sections  representing  specific  industrial  sectors. 
Section  chairs  and  vice  chairs  are  elected  officers.  Section  activities  include  the 
following,  when  appropriate. 

•  Sharing  data,  including  supplier  quality  and  process  or  commodity 
assessment  information 

•  Developing  and  adopting  standard  assessment  criteria  based  on  the  quality 
systems,  processes,  and  commodity  specifications  used  within  that  industry 

•  Pooling  assessment  activities  of  shared  suppliers,  eliminating  redundant 
effort,  or  conducting  joint  assessments  of  large,  critical  suppliers,  reducing  a 
single  member’s  effort  and  cost 

•  Developing  and  adopting  training  requirements,  standards,  and  programs  for 
their  auditors 

•  Developing  standards  for  auditor  qualification  and  certification 

•  Agreeing  to  certify  their  auditors  and  assessors  to  the  developed  criteria 

•  Sharing  supplier  base  management  practices  and  techniques  and  encouraging 
benchmarking 

The  Aerospace/Marine  Systems  section,  which  has  many  defense  contractor 
members,  has  a  Source  Certification  Committee  that  is  responsible  for  preparing  and 
revising  the  Supplier  Quality  System  Evaluation  Checklists  to  assure  standardized 
methods.  Each  supplier  must  be  reevaluated  aimually  to  remain  in  the  data  base,  and  an 
on-site  survey  must  be  conducted  every  3  years.  They  recognize  that — 

In  accepting  supplier  evaluations  (audits/svu^eys)  performed  by  others,  the 
person(s)  who  accomplish  the  evaluation  must  have  established  integrity 
and  credibility  through  demonstrated  proficiency.  To  support  a  standard 
level  of  performance  and  adherence  to  prescribed  procedures,  it  is 
fundamental  that  minimum  requirements  for  aerospace  quality  system 
evaluation  personnel  be  defined.  A  baseline  standard  that  outlines  auditor 
qualification/certification  is  described  in  the  Systems  Procedure  Manual 
for  the  Section  to  enable  members  and  their  customers  to  mutually  accept 
another  member’s  survey  results.  These  minimum  requirements  are  not 
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solely  directed  at  various  personnel  characteristics  but  also  at  the  training 

and  records  for  verification.^ 

Auditors  must  accrue  10  of  a  possible  17  credits  (described  below),  demonstrate 
communication  skills,  and  satisfy  other  requisites  such  as  written  and/or  oral  exams,  audit 
participation,  and  auditor  training  courses.  Credits  accrue  from — 

•  Education:  four  credits  maximum,  based  on  type  of  degree  achieved 

•  Experience:  nine  credits  maximum,  based  on  professional,  quality  assurance, 
and  quality  auditing  experience 

•  Professional  accomplishment:  two  credits  maximum,  based  on  certification 
or  registration  achieved 

•  Management  evaluation:  two  credits  maximum,  based  on  auditor’s 
employer’s  evaluation. 

b.  Supplier  Performance  Information  Network 

The  corporation  maintains  a  C.A.S.E.  Data  Center,  which  is  the  computer  system 
for  collecting,  collating,  publishing,  and  distributing  supplier  information,  including  the 
Supplier  Performance  Information  Network  (SPIN).  SPIN,  accessed  over  the  internet, 
provides  interactive  access  to  the  data  base  of  suppliers  shared  by  the  C.A.S.E.  members. 
The  data  base  includes  product  designation  (simple  or  complex),  identification  of  the 
quality  system  evaluated,  supplier  system  qualifications  by  product/process  codes, 
indication  of  an  on-site  survey  or  performance  update,  the  sustaining  member  who 
performed  the  evaluation,  and  when  the  audits  or  assessments  were  performed.  The 
assessment  results  detail  the  system  level  or  process  technique  for  which  the  supplier  was 
approved.  SPIN  is  available  only  to  authorized  users  who  are  employees  of  a 
participating  C.A.S.E.  member  or  supplier.  Security  is  maintained  through  five  levels  of 
access.  Once  authorized  users  have  access,  they  can  view  profile  information  about  their 
company;  profile  information  of  other  C.A.S.E.  member  companies;  and  archived, 
submitted,  or  current  C.A.S.E.  supplier  profile  or  assessment  information.  Users  can  then 
submit  a  request  to  add,  modify,  or  remove  C.A.S.E.  supplier  profile  or  assessment 
information.  They  can  also  submit  a  request  to  modify  the  list  of  supplier  capabilities 
that  they  use. 


2  Aerospace/Marine  Systems  Procedure,  17  October  17  1995. 
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SPIN  also  provides  access  to  demographic  information  from  suppliers  of  member 
industries.  These  suppliers  are  considered  to  be  approved  sources  when  a  registered 
member  has  assessed  them.  C.A.S.E.  makes  it  clear,  however,  that  “lack  of  approval 
should  not  be  considered  a  reason  not  to  utilize  the  supplier  as  an  industry  source. 
C.A.S.E.  member  goals  include  the  desire  to  approve  all  shared  suppliers.”^ 

The  Chair  of  the  Electronics  and  Computer  Section  of  C.A.S.E.  describes  the 
following  benefits  of  sharing  in  a  supplier  performance  data  base:^ 

•  Access  to  hundreds  of  supplier  performance  assessments 

•  Greatly  increased  knowledge  of  supplier  business  and  technical  process  and 
performance 

•  Assessment  expenses  minimized 

•  Greater  in-depth  assessments 

•  Standardized  assessments 

•  No  lost  or  misplaced  assessment  documentation 

•  Quick  identification  of  key  company  and  supplier  contacts 

•  Quick  connection  to  all  member  companies  and  associated  supplier  profiles 
and  home  pages 

•  Opportunity  to  work  with  C.A.S.E.  membership  to  enhance  their  data  base 
strategy. 

c.  Reciprocal  Agreements 

In  the  winter  of  1997,  C.A.S.E.  and  the  Naval  Sea  Logistics  Center  Detachment 
Portsmouth  (NAVSEALOGCENDETPTSMH),  the  Naval  Material  Quality  Assessment 
Office  (NMQAO),  signed  a  Memorandum  of  Agreement  (MOA)  to  form  a  partnership  to 
exchange  contractor  data  between  C.A.S.E.  and  the  Navy’s  RedA^ellow/Green  program 
(see  section  IV.D).  Sharing  data  bases  is  a  type  of  recognition  by  our  definition.  Boeing 
also  has  this  type  of  arrangement  with  C.A.S.E.  The  following  industry  best  practice 
illustrates  TRW’s  use  of  the  C.A.S.E.  register  for  full  reciprocity  by  the  definition  we  use 
in  this  report. 


^  C.A.S.E.  homepage,  http://www.caseinc.org 

^  Bruce  K.  Short,  Engineering  Manager  of  Digital  Equipment  Corporation  and  Chair  of  Electronics  and 
Computer  Section  of  C.A.S.E.,  white  paper,  “Benefits  from  Sharing  in  a  Supplier  Performance  Data 
Base.” 
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TRW’s  approach  to  supplier  certification  is  based  directly  on  the  ISO  9000 
series.  Indeed,  just  like  ISO  9000,  TRW  has  created  a  quality  assurance 
series  rather  than  an  individual  standard.  This  QSA-9000  series  consists 
of  three  versions,  QSA-9001,  QSA-9002,  and  QSA-9003,  which  deal  with 
different  functional  areas.  Suppliers  complete  the  relevant  QSA-9000 
checklist  indicating  their  current  level  of  compliance  with  the  established 
criteria  (the  20  basic  ISO  9(XX)  criteria).  Corrective  actions  are  to  be 
designed  and  implemented  for  any  noted  deficiencies.  In  addition  to 
having  suppliers  go  through  this  QSA-9000  assessment  process,  TRW 
maintains  a  database  of  the  results  called  the  Quality  Assurance  Supplier 
Directory.  A  supplier’s  compliance  to  QSA-9000  is  proven  through  one 
of  three  ways,  although  TRW  always  reserves  the  right  to  require 
additional  technical  and  product  capability  audits:  1)  certification  of  ISO 
9000  compliance  by  a  third  party  registrar;  2)  inclusion  in  the  C.A.S.E. 
register;  or  3)  determination  of  QSA-9000  compliance  through  a  formal 
audit  or  self-assessment. 

The  Air  Carrier  section  of  C.A.S.E.  has  had  the  greatest  success  in  reducing 
redundant  audits.  C.A.S.E.  identifled  roughly  4,300  redundant  audits  among  the  5,200 
scheduled  audits  of  the  Air  Carrier  members.  The  Air  Carriers  section  reduced  the 
number  of  audits  among  the  members  to  just  over  500  for  1997.  However,  unlike  defense 
contractors  and  the  Aerospace  and  Marine  Section  members.  Air  Carriers  provide 
essentially  one  service  and  share  the  burden  of  a  common  FAA-mandated  standard  for 
supplier  surveillance.  For  the  Aerospace  and  Marine  Section  members,  the  task  of 
reducing  redundant  audits  is  somewhat  more  complicated:  They  make  many  different 
products  for  both  government  and  commercial  customers  and  their  suppliers  work  to 
many  different  quality  system  standards.  Their  goal  of  reducing  redundant  audits  is, 
however,  the  same  as  for  the  Air  Carriers.^ 

2.  National  Aerospace  and  Defense  Contractors  Accreditation  Program 

The  National  Aerospace  and  Defense  Contractor  Accreditation  Program 
(NADCAP)  is  an  industry-driven  supplier  accreditation  program  developed  by 
representatives  from  aerospace  prime  contractors  and  government  agencies  with  the 
support  of  their  supplier  base.  NADCAP’ s  purpose  is  to  reduce  the  number  of  redundant 
quality  systems  audits  that  are  being  performed  on  suppliers. 


5  “The  Case  SPIN  database:  A  who’s  who  of  quality  audits,”  Purchasing,  17  July  1997,  pp.  39-42. 


a.  Organization  Description 

The  Performance  Review  Institute  (PRI),  a  nonprofit  trade  association  affiliated 
with  the  Society  of  Automotive  Engineers  (SAE),  administers  NADCAP.  PRI  was 
founded  for  the  purpose  of  developing  and  advancing  conformity  assessment  programs  to 
raise  special  process  and  quality  standards  on  an  industry-wide  basis  while  minimizing 
redundancy  in  auditing  and  generally  improving  the  process. 

Membership  categories  include  subscribers,  who  are  NADCAP  users  with  voting 
privileges  in  all  activities;  government  agency  representatives,  who  have  voting 
privileges  in  all  activities;  associate  prime  members,  who  can  participate  in  meetings  but 
have  no  voting  privileges;  and  suppliers,  who  participate  in  developing  and  revising  audit 
criteria  and  c^  vote  on  non-accreditation  matters. 

Processes  for  which  task  groups  have  been  formed  include  aerospace  quality 
systems,  chemical  processes,  coatings,  distributors,  fluid  system  components,  fasteners, 
heat  treating,  materials  testing  laboratories,  nondestructive  testing,  sealants,  welding,  and 
the  Fastener  Quality  Act,  for  which  the  National  Institute  for  Standards  and  Technology 
(NIST)  has  found  NADCAP  to  be  in  compliance  for  accreditation  of  fastener  testing 
laboratories.  PRI/NADCAP  has  its  own  Quality  Manual  that  meets  the  intent  of  ISO 
9001. 

Although  PRI  staff  auditors  perform  the  audits,  the  task  groups  decide  on  the 
qualification  requirements  for  the  auditors  and  make  the  final  decision  on  the  contractor 
accreditation.  All  NADCAP  AS90(X)  auditors  must  complete: 

•  ISO-9000  training  (ISO  certification  as  lead  auditor  not  required) 

•  AS9000  training 

•  ISO-l(X)l  1-2  requirements 

—  Work  experience:  4  years  of  full  time  experience,  relative  to  auditing 
—  Quality  experience:  minimum  2  years  of  quality-related  experience 
—  Auditing  experience:  minimum  five  audits  within  last  3  years 

The  Aerospace  Quality  System  task  group  is  discussing  adding  a  requirement  for  all 
auditors  to  interview  with  the  task  group  itself.  Auditor  qualification  options  include  1) 
certified  by  the  RAB,  2)  NADCAP  (Aerospace  Quality  System)  qualified  to  RAB 
criteria,  or  3)  NADCAP  certified  to  NADCAP  criteria. 
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b.  NADCAP  Memorandum  of  Understanding  with  AECMA-CERT 

The  European  Association  of  Aerospace  Industries  (AECMA)  comprises  national 
aerospace  associations  of  the  United  Kingdom,  France,  Belgium,  Italy,  Spain,  and 
Sweden  as  well  as  the  largest  European  aerospace  companies.  AECMA-CERT  is  an 
affiliated  association  to  certify  conformity  to  European  standards  (EN).  The  initial 
meeting  between  PRI/NADCAP  and  AECMA-CERT  took  place  in  Turin,  Italy,  in  June 
1995,  with  additional  meetings  following  in  March  1996.  The  memorandum  of 
understanding  (MOU),  signed  in  September  1996,  is  to  explore  the  potential  for 
certification  reciprocity.  Once  accomplished,  reciprocity  would  mean  that  suppliers 
certified  by  NADCAP  would  be  able  to  conduct  business  in  Europe  without  duplicating 
the  time-consuming  and  expensive  audit  process. 

A  meeting  was  held  on  6  November  1997  to  follow  up  on  the  MOU  by  outlining 
the  concerns  of  AECMA-CERT  and  PRI/NADCAP  members,  defining  the  scope  of 
work,  chartering  the  coordinating  group,  and  developing  a  joint  communique  to  inform 
the  aerospace  industry  of  the  work  to  date. 

c.  NADCAP  Memorandum  of  Understanding  with  QUALIFAS 

Like  AECMA-CERT,  the  Quality  of  Procurements  for  French  Aerospace 
Industries  (QUALIFAS)  also  audits  European  companies  to  a  variety  of  standards.  The 
QUALIFAS  and  PRI/NADCAP  have  signed  an  MOU  for  evaluating  each  other’ s  quality 
manuals  and  auditing  and  supplier  approval  processes.  This  process  is  ongoing. 

C.  STANAG  4107— MUTUAL  ACCEPTANCE  OF  GOVERNMENT  QUALITY 
ASSURANCE 

The  North  Atlantic  Treaty  Organization  (NATO)  has  a  standardization  agreement 
whereby  NATO  countries  request  and  accept  the  quality  assurance  services®  of  one 
another’s  designated  quality  authorities  in  order  to  assure  the  quality  of  military  materiel 
and  services  produced  in  NATO  countries.  This  agreement — STANAG  4107,  Mutual 
Acceptance  of  Government  Quality  Assurance — details  the  terms  and  procedures  for 
cooperation  between  NATO  members  to  provide  for  the  quality  assurance  of  their 
defense  suppliers. 


®  STANAG  4107  defines  Government  Quality  Assurance  as  “the  process  by  which  the  appropriate 
national  authorities  establish  competence  that  contractual  requirements  relating  to  quality  are  met.” 
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The  framework  of  ST  AN  AG  4107  provides  a  process  through  which  a  purchasing 
NATO  country  may  request  that  the  assigned  authority  in  a  manufacturing  NATO 
country  perform  the  required  quality  assurance  services  on  its  behalf.  The  NATO  country 
in  which  the  supplier  resides,  therefore,  evaluates  the  contractor’s  quality  assurance 
procedures  to  assure  the  purchasing  NATO  country  that  the  contractor  will  produce  and 
deliver  quality  products  and  services,  which  comply  with  contractual  requirements. 
STANAG  4107  is  used  when  the  contract  places  a  high  priority  on  the  verification  of 
quality  and  stipulates  that  it  must  be  done  before  the  products  or  services  are  received. 

The  procedure  for  performing  quality  assurance  services  for  another  NATO 
country  begins  when  the  purchasing  country  submits  a  formal  Request  for  Quality 
Assurance  form  to  the  appropriate  authority  in  the  manufacturing  coimtry.  The  initial 
request  details  the  contract’s  specifications,  drawings,  and  any  additional  quality 
requirements  necessary  to  perform  a  quality  assurance  evaluation.  The  designated 
government  authority  in  the  manufacturing  country  assesses  the  supplier  on  the  basis  of 
the  quality  requirements  indicated  in  the  contract  and  makes  the  necessary  provisions  for 
any  identified  nonconformities  to  be  corrected.  If  at  any  time  the  government  authority  in 
charge  of  performing  quality  assurance  determines  that  a  supplier  is  so  deficient  as  to 
hinder  further  evaluation,  the  purchasing  country  will  be  informed  of  this  status  and 
provided  with  any  data  relating  to  the  identified  deficiencies.  If  a  supplier’s  quality  has 
been  proven,  however,  the  manufacturing  government  authority  provides  the  purchasing 
country  with  a  certificate  of  the  supplier’s  conformity.  Although  the  manufacturing 
country  performs  this  quality  assurance  service  on  behalf  of  a  purchasing  country,  the 
purchasing  country  reserves  the  right  to  visit  a  supplier  at  any  point  during  a  contract,  as 
arranged  through  the  manufacturing  country’s  authority. 

NATO  countries  may  define  their  level  of  acceptance  of  STANAG  4107  in  three 
ways:  1)  ratification,  2)  implementation,  and  3)  reservation.  Ratification  occurs  when 
NATO  countries  indicate  their  acceptance  of  the  standardization  agreement’s  content. 
Implementation  occurs  when  a  NATO  member  country  carries  out  its  responsibilities  as 
outlined  in  the  standardization  agreement.  Finally,  a  NATO  member  country  may  note  a 
reservation  or  stipulation  that  it  cannot  adhere  to  a  particular  part  of  the  standardization 
agreement.  The  U.S.  maintains  a  reservation  that  states  that  the  quality  assurance 
services  performed  by  the  U.S.  government  authority  will  not  be  free  of  charge  unless 
previously  negotiated  in  an  agreement  between  the  United  States  and  the  requesting 
NATO  country. 
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As  a  rule,  the  government  quality  assurance  service  is  provided  without  charge  to 
the  purchasing  NATO  country,  as  any  expenses  incurred  are  covered  by  the  contracting 
parties.  This  quality  assurance  arrangement  fosters  a  cooperative  relationship  between 
the  two  NATO  countries’  authorities  with  respect  to  quality  and  quality-related  issues. 
The  quality  assurance  that  takes  place  is  based  not  on  one  specific  standard,  but  on 
various  individual  specifications  as  detailed  in  each  contract.  Each  country  must  have  a 
high  level  of  cooperation  and  trust  in  the  other’s  quality  assurance  capabilities  since  the 
purchasing  country  is  entrusting  its  role  of  assuring  the  quality  of  its  suppliers  to  the 
manufacturing  country’s  authority.  Therefore,  it  is  important  that  each  of  the  NATO 
countries’  authorities  convey  their  competence  in  conducting  quality  assurance  services 
based  not  only  on  one  standard’s  criteria,  but  also  on  the  various  requirements  that  pertain 
to  any  given  contract. 

D.  CURRENT  DOD  POLICY 

DoD  policymakers  have  recognized  that  substantial  costs  are  associated  with 
supplier  control  and  that  routine  supplier  audits  conducted  independently  by  each  prime 
contractor  can  be  duplicative,  repetitive,  and  costly.  Dr.  Paul  Kaminski,  in  his  speech 
“Standards  and  the  Single  Process,”  recognized  NADCAP  as  a  possible  contribution  to 
the  single  process  initiative: 

By  significantly  reducing  redundant  audits  and  multiple  processes,  costs 
and  cycle  time  have  been  reduced  by  as  much  as  50  percent.  Certainly,  if 
industry  can  unite  behind  a  single  process  standard  and  certification 
procedure,  it  makes  a  strong  case  for  our  consideration  under  the  single 
process  initiative. 

The  Office  of  Management  and  Budget  (0MB)  Circular  119,  “Federal 
Participation  in  the  Development  and  Use  of  Voluntary  Consensus  Standards  and  in 
Conformity  Assessment  Activities,"  is  currently  being  revised  to  align  with  Public  Law 
104-113,  the  National  Technology  Transfer  Act  of  1995.  Paragraph  11  of  that  revised 
document  states: 

Conformity  Assessment.  Section  12(b)  of  P.L.  104-113  requires  NIST  to 
coordinate  Federal,  State,  and  local  standards  activities  and  conformity 
assessment  activities  with  private  sector  standards  activities  and 
conformity  assessment  activities,  with  the  goal  of  eliminating  unnecessary 
duplication  and  complexity  in  the  development  and  promulgation  of 
conformity  assessment  requirements  and  measures.  To  ensure  effective 
coordination,  NIST  shall  issue  guidance  to  the  agencies. 
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Such  action  should  also  enhance  opportunities  for  reciprocal  agreements  among 
the  federal  agencies  and  with  private  organizations  such  as  NADCAP  and  C.A.S.E. 
Some  participation  is  already  taking  place.  For  example,  the  Defense  Logistics  Agency 
(DLA)  holds  a  “Quality  Day”  every  few  months  for  its  customers.  During  the  October 
1997  Quality  Day,  DLA  also  invited  two  members  from  C.A.S.E.  to  explain  the  benefits 
of  the  reduction  in  audits  that  their  program  can  bring  to  the  buying  activities. 

The  Defense  Contract  Management  Command  (DCMC),  which  performs  an 
oversight  function  in  assessing  the  quality,  technical,  and  production  compliance  of  DoD 
contractors,  has  had  a  policy  since  18  November  1996  that  advises  taking  third  party 
audit  results  into  account  when  appropriate.  Since  that  time,  DCMC  policy  has  stated 
that  duplicative  auditing  efforts  shall  not  be  undertaken  if  data  already  exist  to  document 
that  a  contractor’s  quality  meets  a  particular  contract’s  quality  requirements.  “DCMC 
personnel  shall  evaluate  contractor  quality  systems  for  compliance  with  contract 
requirements  using  existing  data.”  General  Drewes  also  issued  DCMC  Memorandum 
No.  97-37,  “Management  Council  Reduction  of  Redundant  Supplier  Audits,”  which 
recognized  both  C.A.S.E.  and  NADCAP:  “We  welcome  and  applaud  credible  third  party 
programs  and  industry  efforts  to  self-control.”  This  memo  encourages  DCMC  personnel 
to  work  with  contractors  within  their  local  management  councils  to  actively  look  for  ways 
to  reduce  or  eliminate  redundant  prinae  contractor  audits  of  suppliers,  by  relying  on  third 
party  or  industry  approvals  where  feasible.  When  evaluating  quality  systems  the  policy 
directs  that — 

DCMC  persotmel  shall  evaluate  contractor  quality  systems  for  compliance 

with  contractor  requirements  using  existing  data  (e.g.,  audit  reports)  from 

credible  first,  second,  or  third  party  audits. 

The  government’s  formal,  documented  policy  is  to  avoid  duplicating  audits 
already  performed  when  these  audits  and  their  auditors  can  be  verified.  DCMC, 
however,  still  requires  a  DCMC  quality  audit  when — 

•  Data  from  a  previous  audit  are  inadequate  or  unavailable 

•  The  customer  specifically  requests  a  DCMC  audit 

•  There  is  an  indication  that  a  contractor’s  performance  is  not  compatible  with 
the  contract  requirements 

•  The  contractor  has  undertaken  major  modifications  of  its  quality  system 
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Any  audits  undertaken  by  DCMC  should  be  confined  to  those  areas  specifically  identified 
by  the  customer  as  important  to  a  contractor’s  quality  compliance  with  a  contract’s 
requirements. 

DCMC  utilizes  the  ISO  9000  or  ANSI/ASQ  Q9000  series  of  standards  in 
conducting  its  audits.  DCMC  auditors  use  a  checklist  of  questions,  based  on  the  20 
criteria  of  ISO  9000,  as  a  template  to  evaluate  the  quality  systems  of  government 
contractors.  When  the  DCMC  audit  is  complete,  the  auditors  present  the  audit  results  to 
the  contractor  and  customer.  If  the  contractor  is  not  in  compliance  with  the  contract’s 
quality  management  system  criteria,  the  contractor  is  asked  to  implement  corrective 
actions.  Once  all  corrective  actions  (if  any)  have  been  made  and  the  contractor  is 
determined  to  be  in  compliance  with  the  contract’s  quality  requirements,  the  contractor  is 
considered  to  be  qualified. 

DCMC  encourages  and  benefits  from  the  audit  work  of  industry  organizations 
such  as  C.A.S.E.  and  NADCAP  because  DCMC  can  often  rely  on  their  audit  reports, 
thereby  reducing  or  eliminating  the  need  for  a  DCMC  audit.  To  date,  however, 
membership  in  these  organizations  is  not  yet  widespread  across  all  industry  sectors. 
DCMC  manages  more  than  360,000  prime  contracts  worth  more  than  $900  billion  at 
more  than  23,000  contractors  throughout  the  world.  Any  substantial  gains  to  the 
government  from  a  relationship  with  these  industry  organizations  is,  therefore,  hindered 
by  their  currently  limited  size  and  scope. 

E.  PROPOSAL  BY  GIQLP 

The  Government  and  Industry  Quality  Liaison  Panel  (GIQLP)  consists  of  a 
number  of  members  from  participating  government  agencies  and  industry  associations. 
Their  goal  is  to  advance  quality  practices  and  remove  non-value  added  costs  hidden  in 
restrictive  requirements  and  redundant  oversight  and  inspection  processes.  The  Panel 
has  recommended  a  process  for  recognition  and  reciprocity  in  their  recent  document 
Quality  Management  Systems  Guide. 

Mutual  Recognition  of  a  Supplier  Quality  Management  System.  An  on¬ 
site  supplier  quality  management  system  assessment  by  one  participating 
procuring  activity  or  its  agent  should  be  recognized  and  accepted  by  the 
other  procuring  activities  as  adequate  evidence  that  the  system  was  found 
to  comply  with  BQMS  [Basic  Quality  Management  System]  criteria  at  that 
time.  Multiple  reviews  and  duplicate  demands  of  a  supplier  or  supplier  by 
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several  agencies  should  be  reduced  to  the  maximum  practicable  extent 
through  assessment  reciprocity  or  cross-servicing  arrangements. 

The  one  constant  across  government  agencies  or  industry  groupings  that 
enables  mutual  recognition  of  a  supplier’s  quality  management  system  is  a 
basic  quality  system  based  on  the  elements  of  ISO  9001.  There  may  be 
further  opportunity  for  mutual  recognition  within  an  agency  or  industry 
group  of  commonly  agreed  upon  terms  and  conditions  required  of 
suppliers. 

Reciprocity  for  Quality  System  Audits.  To  enable  reciprocity  and  mutual 
recognition  of  a  supplier’s  quality  management  system,  at  the  conclusion 
of  an  on-site  assessment  each  procuring  activity  or  its  agent  will  leave  the 
supplier  a  copy  of  the  assessment  criteria  (i.e.,  checklist)  employed,  the 
assessment  results,  and  a  statement  of  qualification  of  the  performing 
assessors.  Third  party  assessors  will  also  be  requested  to  provide  similar 
information  to  the  supplier. 

Another  procuring  activity  may  then  request  of  the  supplier  objective 
evidence  of  quality  system  assessments  by  other  customers,  agents,  or 
third  parties.  The  requesting  customer  will  evaluate  the  prior  assessment 
to  determine  suitability  to  satisfy  assessment  requirements.  The  customer 
may  then  consider  the  supplier’s  quality  system  to  be  qualified  based  on 
the  evidence  provided  by  the  supplier,  or  determine  that  another  or  a 
partial  assessment  is  required  and  Aen  cany  it  out  accordingly. 

Advantage  gained  is  based  on  a  review  of  the  documentation  of  the 
assessment  performed,  with  the  audit  requirement  determined  to  be  fiilly 
satisfied  and  “signed-off’  (full  reciprocity)  or  that  the  assessment  has 
partially  satisfied  requirements  and  a  limited  assessment  will  be  performed 
(partial  reciprocity).  The  customer  always  has  the  choice  of  accepting  the 
validity  of  a  previously  performed  audit,  or  doing  a  complete  or  partial 
audit  of  the  supplier’s  qudity  management  system 

In  this  GIQLP  definition,  the  essence  of  reciprocity  and  recognition  is  customers 
agreeing  to  honor  each  other's  assessments  and  audit  results.  This  definition  is  difierent 
fi-om  that  used  in  this  report  and  may  not  distinguish  the  two  sufficiently  or  afford  an 
opportunity  to  truly  impact  the  subcontractor  base  assessment  and  audit  process.  We 
recognize  that  industry  contractors  are  somewhat  hesitant  to  recognize  someone  else’s 
audits.  Industry  also  would  not  want  another  bureaucratic  government  organization  to 
coordinate  the  reciprocal  audit  process.  This  is  why  organizations  such  as  C.A.S.E.  and 
NADCAP  are  welcomed  by  DCMC  policy. 
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VI.  CONCLUSIONS  AND  RECOMMENDATIONS 


A.  STUDY  RESULTS 

There  are  many  reasons  why  a  supplier  would  want  to  win  an  award  or  become 
certified  or  registered  to  a  standard.  Foremost  is  the  prospect  of  increased  business 
opportunities.  Most  qualification  programs  work  as  thresholds — either  the  supplier  meets 
the  standard  of  the  program  required  by  the  customer  or  the  supplier  doesn’t  do  business 
with  that  customer.  The  prestige  that  comes  with  winning  a  quality  award  also  usually 
results  in  new  business  in  the  conunercial  sector.  For  defense  contracts,  audits  by  DCMC 
to  determine  that  the  supplier  meets  an  ISO-like  system  tailored  to  the  needs  of  the 
contract  are  required.  New  acquisition  rules  now  let  DoD  buying  authorities  consider  the 
past  performance  of  a  contractor — quality,  schedule,  and  cost  performance — ^in  source 
selection. 

The  problems  with  audits  for  a  supplier  come  when  different  customers  require 
different  types  of  audits  under  different  types  of  certification/registration/accreditation 
programs.  Many  of  these  audits  are  similar  in  nature  and  content,  as  illustrated  by  our 
comparison  tables  in  Chapters  n  and  m.  Similarity  among  programs  is  a  result  of  using 
the  20  elements  of  the  ISO  9000  standards  as  their  basis.  But  each  audit  costs  the 
supplier  and  the  customer  substantial  amounts  of  time  and  money.  Data  we  collected 
shows  that  ISO  9000  registration  alone  can  cost  a  company  as  much  as  $30,000  initially 
and  up  to  $5,000  every  6  months  for  reassessment  to  maintain  the  registration.* 

Shared  data  bases  and  reciprocal  agreements  among  companies  in  formal 
organizations  such  as  C.A.S.E.  and  NADCAP  have  saved  their  member  companies  both 
time  and  money  as  the  following  examples  illustrate.  One  C.A.S.E.-member  company 
reports  that  it  saves  an  average  of  100  audit  surveys  per  year.  Each  survey  entails  about 
32  hours  of  labor  and  travel  cost,  which  works  out  on  average  to  $5,000  per  survey.  A 
large  prime  contractor  company  reports  net  savings  of  $1.6  million  per  year  due  to  its 


*  Steven  M.  Terronez,  “The  Contractor  Performance  Certification  Program,”  Army  RD&A,  Sept-Oct 
1997. 
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NADCAP  membership.  Another  prime  reports  savings  of  over  200  audits  per  year. 
NADCAP  supplier  companies  report  savings  of  about  40%  fewer  audits  p)er  year.  Their 
business  volume  has  increased  $750,000  to  $2  million. 

Despite  such  reported  savings,  trust  remains  an  issue  for  most  primes — they  prefer 
to  do  their  own  quality  system  audits  rather  than  rely  on  those  done  by  one  of  their 
competitors,  and  they  worry  about  the  legal  repercussions  of  sharing  their  audit  results  of 
suppliers  with  other  customers.  C.A.S.E.  members  determined  that  their  organization 
presents  no  legal  liability,  as  demonstrated  by  the  fact  that  it  has  operated  for  over  30 
years  without  successful  legal  challenges.  The  trick  is  to  have  an  arrangement  whereby 
the  participating  original  equipment  manufacturers  (OEMs)  agree  on  the  processes  for  the 
audits,  the  auditor  qualifications,  and  the  criteria  for  the  audits  (usually  based  on  ISO- 
9000  with  some  sector-specific  additions).  Then  the  contractors  can  share  nonprejudicial 
data  on  the  process,  completion,  and  pass-fail  results  of  the  audit  without  worrying  about 
legal  repercussions  of  sharing  the  actual  audit  results.  This  is  the  procedure  C.A.S.E. 
uses  and  it  has  been  lawsuit-free  since  its  inception  in  1964.  C.A.S.E.  was  formed  when 
several  prime  contractors  who  shared  many  of  the  same  suppliers  banded  together  to 
derive  a  process  whereby  they  could  reduce  the  number  of  audits  and  assessments  they 
had  to  perform  on  their  suppliers. 

B.  RECOMMENDATIONS 

To  reduce  the  time  and  cost  of  quality  system  audits,  DoD  would  do  well  to  make 
use  of  as  much  industry  data  and  as  many  industry  programs  as  possible.  This  has  been 
DCMC  policy  since  November  1996,  and  we  recommend  that  DoD  continue  along  this 
path.  Because  DoD  cannot  recommend  one  industry  program  over  another,  industry  must 
make  the  determination  of  which  reciprocal  organization  meets  its  needs.  Since  trust  is 
best  built  among  customers  who  share  similar  products,  these  organizations,  or  divisions 
and  sections  within  the  organizations,  should  be  built  around  industry  sectors,  as  the 
sections  of  C.A.S.E.  and  NADCAP  are.  Since  DCMC  deals  with  thousands  of  products 
that  do  not  currently  have  reciprocal  agreements  and  organizations  within  their  sectors,  its 
only  option  is  to  do  its  own  audits  under  the  current  policy  of  using  any  other  audit 
information  available  from  the  supplier.  DoD  would  do  well  to  encourage  the  defense 
and  commercial  industry  to  look  at  the  benefits  of  forming  reciprocal  relationships  for 
their  quality  audits.  When  such  arrangements  become  best  commercial  practice,  DoD  can 
make  full  use  of  them.  As  with  other  processes  under  the  Single  Process  Initiative,  the 
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greatest  benefit  comes  when  streamlining  practices  not  only  affect  prime  contractors  but 
also  pervade  the  supplier  base.  We  repeat  here  the  recommendations  of  the  gear  industry 
report  on  the  Common  Quality  Certification  System  (CQCS): 

Finally,  for  a  CQCS  to  be  truly  successful,  it  must  be  applicable  at  all 
supplier  levels.  Therefore,  primes  need  to  undertake  efforts  to  reduce  the 
redundancies  and  foster  reciprocity  in  the  auditing  and  certification  of  their 
own  suppliers.  Also,  the  primes  need  to  agree  on  criteria  and  encourage 
reciprocity  in  the  audit  and  certification  of  their  suppliers,  otherwise,  a 
CQCS  will  provide  inconsistent  benefits  to  the  industry  as  a  whole. 
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For  each  program  in  this  appendix,  criteria  are  listed  in  three  categories — ^process 
criteria,  auditor  criteria,  and  results  criteria.  Process  criteria  are  those  attributes  of  the 
process  one  must  go  through  to  receive  registration  to  a  particular  program  or  standard. 
Auditor  criteria  are  the  qualification  requirements  for  the  auditors  of  a  specific  program 
or  standard.  Results  criteria  are  those  characteristics  against  which  the  quality  system  is 
judged. 

ISO  9000 
Process  Criteria 

Preassessment  Phase 

•  Company,  consultant,  or  registration  agency  assesses  the  existing  quality 
system 

•  Company  designs,  modifies,  and/or  implements  a  quality  system  that 
addresses  ISO  9000  criteria 

Assessment  Phase 

•  Registration  audit  (comprehensive  audit  of  the  organization)  performed  by 
third  party  registration  agency  (registrar) 

•  Any  cases  of  noncompliance  are  presented 

•  Third  party  registration  agency  may  conduct  additional  audits/reviews  to 
determine  if  nonconformance  issues  have  been  addressed 

•  Certificate  of  conformance  to  ISO  9000  standard  presented  to  the  company  or 
organization 

Postassessment  Phase 

•  Periodic  surveillance  audits 

•  Certification  generally  expires  in  3  years 
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Auditor  Criteria 

•  Auditor  must  have  completed  a  secondary  education  and  possess 
demonstrated  oral  and  written  communication  skills 

•  Auditor  must  have  participated  in  training  to  ensure  auditing  competence 
based  on  a  demonstrated  understanding  of  the  following:  standards; 
examination,  questioning,  evaluation,  and  reporting  techniques;  and  the  skills 
necessary  for  managing  an  audit 

•  Auditor  must  have  at  least  4  years  of  full-time  work  experience,  2  of  which 
involve  quality  assurance 

•  Auditor  must  gain  experience  in  the  auditing  process  by  participating  in  at 
least  4  audits  over  a  minimum  of  20  days 

•  Auditor  must  be  proven  to  possess  personality  attributes,  including  maturity, 
open-mindedness,  solid  judgment,  analytical  skills,  tenacity,  perceptiveness, 
the  ability  to  comprehend  complex  situations  from  varying  perspectives,  and 
the  ability  to  understand  a  single  unit’s  role  in  the  whole  organization 

•  Auditor  must  demonstrate  the  knowledge  and  use  of  the  management  skills 
necessary  in  performing  audits 

•  Auditors  must  have  their  competence  in  current  quality  system  standards  and 
the  auditing  process  and  methods  reviewed  through  a  performance  review 
every  3  years,  with  training  being  prescribed  as  necessary 

Results  Criteria 

1.  Management  responsibility 

1.1  Quality  policy 

1.2  Organization 

1 .2. 1  Responsibility  and  authority 

1.2.2  Resources 

1.2.3  Management  representative 

1.3  Management  review 

2.  Quality  system 

2.1  General 

2.2  Quality  system  procedures 

2.3  Quality  planning 

3.  Contract  review 

3.1  General 

3.2  Review 

3.3  Amendment  to  contract 

3.4  Records 
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4.  Design  control 

4.1  General 

4.2  Design  and  development  planning 

4.3  Organizational  and  technical  interfaces 

4.4  Design  input 

4.5  Design  output 

4.6  Design  review 

4.7  Design  verification 

4.8  Design  changes 

5.  Document  and  data  control 

5.1  General 

5.2  Document  and  data  approval  and  issue 

5.3  Document  and  data  changes 

6.  Purchasing 

6.1  General 

6.2  Evaluation  of  subcontractors 

6.3  Purchasing  data 

6.4  Verification  of  purchased  product 

7.  Control  of  customer-supplied  product 

8.  Product  identification  and  traceability 

9.  Process  control 

10.  Inspection  and  testing 
10.1.  General 

10.2  Receiving  inspection  and  testing 

10.3  In-process  inspection  and  testing 

10.4  Final  inspection  and  testing 

10.5  Inspection  and  test  records 

1 1 .  Control  of  inspection,  measuring  and  test  equipment 

11.1  General 

11.2  Control  procedure 

12.  Inspection  and  test  status 

13.  Control  of  nonconforming  product 

13.1  General 

13.2  Review  and  disposition  of  nonconforming  product 

14.  Corrective  and  preventive  action 

14.1  General 

14.2  Corrective  action 

14.3  Preventive  action 

15.  Handling,  storage,  packaging,  preservation,  and  delivery 

15.1  General 

15.2  Handling 
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15.3  Storage 

15.4  Packaging 

15.5  Preservation 

15.6  Delivery 

16.  Control  of  quality  records 

17.  Internal  quality  audits 

18.  Training 

19.  Servicing 

20.  Statistical  techniques 

20. 1  Identification  of  need 

20.2  Procedures 

BOEING  Dl-9000 
Process  Criteria 
Preassessment  Phase 

•  Assistance  provided  to  supplier  on  how  to  interface  with  Boeing  during 
Dl-9000  approval  process 

Assessment  Phase 

•  Initial  qualification  audit  of  suppliers’  quality  and  production  and  control 
systems  by  the  procurement  quality  assurance  representatives 

•  Corrective  actions  implemented  by  the  supplier  if  deficiencies  are  discovered 

•  Supplier  conformity  with  Boeing  specific  requirements  (i.e.,  quality  and 
contract  requirements;  CAD  system  to  use;  and  the  execution  of  first  article 
testing)  ensured 

Postassessment  Phase 

•  Maintenance  audits  conducted  by  the  procurement  quality  assurance 
representatives 

•  Follow-up  on  reports  from  Boeing  and  customers  documenting  any  quality 
concerns 

Auditor  Criteria 

•  Procurement  quality  assurance  representatives  from  Boeing 
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Results  Criteria 


Basic  Quality  System 

1.  Management  responsibility 

1.1  Quality  policy 

1.2  Organization 

1 .2. 1  Responsibility  and  authority 

1.2.2  Resources 

1 .2.3  Management  representative 

1 .2.4  Management  representative  notification 

1 .2.5  Delegated  quality  activities 

1.3  Management  review 

2.  Quality  system 

2.1  General 

2.2  Quality  system  procedures 

2.3  Quality  planning 

3.  Contract  review 

3.1  General 

3.2  Review 

3.3  Amendment  to  a  contract 

3.4  Records 

4.1  Design  authority 

5.  Document  and  data  control 

5.1  General 

5.2  Document  and  data  aq)proval  and  issue 

5.3  Document  and  data  changes 

5.3.1  Document  change  incorporation 

6.  Purchasing 

6.1  General 

6.1.1  Responsibility 

6.2  Evaluation  of  subcontractors 

6.3  Purchasing  data 

6.4  Verification  of  purchased  product 

6.4. 1  Supplier  verification  at  subcontractor’s  premises 

6.4.2  Customer  verification  of  subcontracted  product 

6.4.3  Delegation  of  supplier  verification  to  subcontractors 

6.4.4  Right  of  entry 

6.5  Requirements  flowdown 

7.  Control  of  customer — supplied  product 

7.1  Notification  and  au^orization 

8.  Product  identification  and  traceability 
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9.  Process  control 

9. 1  Process  specification  requirements 

9.2  Tooling 

10.  Inspection  and  testing 

10.1  General 

10. 1 . 1  Approved  inspection  and  test  sources 

10.2  Receiving  inspection  and  testing 

10.3  In-process  inspection  and  testing 

10.4  Final  inspection  and  testing 

10.4.1  Use  of  Boeing  digital  data  sets  as  authority  for  design  and/or 
inspection 

10.5  Inspection  and  test  records 

10.5. 1  First  production  article 

10.6  Inspection  options 

1 1 .  Control  of  inspection,  measuring  and  test  equipment 

11.1  General 

11.1.1  Definition 

1 1 .2  Control  procedure 

12.  Inspection  and  test  status 

12.1  Inspection  stamps 

13.  Control  of  nonconforming  product 

13.1  General 

13.2  Review  and  disposition  of  nonconforming  product 

13.2. 1  Material  review  authority 

13.2.2  Regrading  material 

13.2.3  Scrap  material 

13.2.4  Material  review  of  supplier  designs 

13.2.5  Notification 

14.  Corrective  and  preventive  action 

14. 1  General 

14.1.1  Repetitive  nonconformances 

14.2  Corrective  action 

14.2. 1  Corrective  action  response  format 

14.3  Preventive  action 

15.  Handling,  storage,  packaging,  preservation  and  delivery 

15.1  General 

15.2  Handling 

15.3  Storage 

15.3. 1  Configuration  control  of  inventory 

15.3.2  Control  of  excess  inventory 

15.4  Packaging 

15.5  Preservation 
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15.6  Delivery 

15.6.1  Shipping  documents 

16.  Control  of  quality  records 

1 6. 1  Record  retention  and  availability 

17.  Internal  quality  audits 

17.1  Annual  audit 

18.  Training 

18.1  Proficiency  assessment 

19.  Servicing 

20.  Statistical  techniques 

20. 1  Identification  of  need 

20.2  Procedures 

20.3  Acceptance  sampling 

Additional  Criteria 

Advanced  Quality  System 

1.  Determine  key  characteristics.^  A  Boeing  supplier  must  incorporate  key 
characteristics,  whether  identified  by  Boeing  or  through  a  supplier  team  analysis, 
into  its  Advanced  Quality  System  (AQS)  Control  Plan.  Furthermore,  suppliers 
are  required  to  “flowdown”  key  characteristics  to  subcontractors. 

1 . 1  Does  Boeing  provide  key  characteristics? 

1 .2  Collect  data  to  determine  key  characteristics 

1.3  Establish  key  characteristics 

1.4  Document  key  characteristics  and  engineering  specifications  on  AQS 
control  plan 

2.  Provide  evidence  of  variation.  Once  key  characteristics  have  been  identified,  the 
supplier  must  determine  the  most  appropriate  point  in  the  manufacturing  process 
and  the  tools  to  be  used  to  take  measurements.  These  measurements  are  recorded 
on  a  control  chart  in  order  to  track  a  key  characteristic’s  variation.  Suppliers  must 
record  their  control  chart  choice,  Ae  point  in  the  process  at  which  the 
measurements  are  taken,  and  the  frequency  with  which  the  measurements  are 
taken  on  their  AQS  Control  Plan. 

2. 1  Determine  process  steps  where  key  characteristics  are  measured 

2.2  Select  appropriate  control  charts 

2.3  Document  process  steps,  control  charts.  Sample  size,  and  fiequency  on 
AQS  control  plan 

2.4  Collect  measurements  and  maintain  control  charts 


^  “A  key  characteristic  is  a  feature  whose  variation  has  the  greatest  impact  on  the  fit,  performance,  or 
service  life  of  the  finished  product  from  the  perspective  of  the  customer,  ”  Boeing  D 1-9000. 
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3.  Identify  and  control  sources  of  variation.  Suppliers  are  to  use  the  measurements 
plotted  on  a  control  chart  to  determine  whether  a  key  characteristic  is  in  control 
and  calculate  whether  its  capabilities  are  adequate  to  meet  specific  engineering 
specifications.  If  a  key  characteristic  is  discovered  to  be  out  of  control,  the 
supplier  must  identify  the  causes  of  variations  and  take  action  to  eliminate  these 
special,  gage,  and/or  process  variation  causes.  Once  corrective  actions  have  been 
made,  new  measurements  must  be  taken  and  plotted  in  order  to  show  that  the  key 
characteristic  is  once  again  in  control.  All  data  regarding  the  elimination  of 
variations  is  recorded  in  the  supplier’s  quality  system  documentation. 

3. 1  Determine  whether  key  characteristic  is  in  statistical  control 

3.2  Determine  whether  key  characteristic  meets  minimum  capability 

3.3  Assess  whether  special  causes  of  variation  can  be  assigned 

3.4  Remove  special  causes  of  variation 

3.5  Collect  new  measurements 

3.6  Verify  whether  gage  variation  study  has  been  performed  and  documented 

3.7  Perform  gage  variation  study  and  document  results  on  AQS  control  plan 

3.8  Determine  whether  corrective  action  was  taken  on  measurement  system 

3.9  Identify  potential  sources  of  process  variation 

3.10  Correlate  sources  of  process  variation  with  the  key  characteristic 

3.11  Establish  controls  for  key  process  parameters 

3.12  Document  operation,  key  process  parameters,  process  parameter  settings, 
and  control  method  on  AQS  control  plan 

3.13  Update  process  data  base  or  historic^  records 

4.  Process  orientation.  If  many  different  parts  share  the  same  key  characteristics  and 
processes,  the  supplier  may  wish  to  adopt  a  method  of  process  control  to 
minimize  variation.  To  do  so,  a  supplier  must  have  a  solid  understanding  of  the 
relationship  between  process  output  and  process  input  parameters,  which  must 
function  in  control. 

4. 1  Process  output  control 

4.2  Process  input  control 

AEROSPACE  INDUSTRY  AS-9000 
Process  Criteria 

•  Compatible  with  the  ISO  9000  registration  process 
Auditor  Criteria 

•  Dependent  upon  the  validation  process  chosen 
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Results  Criteria 

1.  Management  responsibility 

1.1  Quality  policy 

1.2  Organization 

1.3  Management  review 

2.  Quality  system 

2.1  General 

2.2  Quality  system  procedures 

2.3  Quality  planning 

3.  Contract  review 

3.1  General 

3.2  Review 

3.3  Amendment  to  contract 

3.4  Records 

4.  Design  control 

4.1  General 

4.2  Design  and  development  planning 

4.3  Organizational  and  technical  interfaces 

4.4  Design  input 

4.5  Design  output 

4.6  Design  review 

4.7  Design  verification 

4.8  Design  validation 

4.9  Design  changes 

5.  Document  and  data  control 

5.1  General 

5.2  Document  and  data  approval  and  issue 

5.3  Document  and  data  changes 

6.  Purchasing 

6.1  General 

6.2  Evaluation  of  subcontractors 

6.3  Purchasing  data 

6.4  Verification  of  purchased  product 

6.5  Requirements  flowdown 

7.  Control  of  customer-supplied  product 

8.  Product  identification  and  traceability 

9.  Process  control 

9.1  Process  specification  requirements 

9.2  Tooling 

10.  Inspection  and  testing 

10.1  General 
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10.2  Receiving  inspection  and  testing 

10.3  In-process  inspection  and  testing 

10.4  Final  inspection  and  testing 

10.5  Inspection  and  test  records 

1 1 .  Control  of  inspection,  measuring  and  test  equipment 

11.1  General 

1 1 .2  Control  procedure 

12.  Inspection  and  test  status 

12.1  Acceptance  authority  media 

13.  Control  of  nonconforming  product 

13.1  General 

13.2  Review  and  disposition  of  nonconforming  product 

14.  Corrective  and  preventive  action 

14.1  General 

14.2  Corrective  action 

14.3  Preventive  action 

15.  Handling,  storage,  packaging,  preservation  and  delivery 

15.1  General 

15.2  Handling 

15.3  Storage 

15.4  Packaging 

15.5  Preservation 

15.6  Delivery 

16.  Control  of  quality  records 

16.1  Record  availability 

17.  Internal  quality  audits 

18.  Training 

19.  Servicing 

20  Statistical  techniques 

20. 1  Identification  of  need 

20.2  Procedures 

20.3  Sampling  inspection 

AUTOMAKERS  QS-9000 
Process  Criteria 

Preassessment  Phase 

•  Sharing  of  quality  system  information 

•  Supplier  applies  for  QS-9000  certification  and  provides  necessary 
documentation 
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Assessment  Phase 

•  Registration  audit  by  third  party  registrar 

•  Nonconformities  outlined 

•  Series  of  audits  by  third  party  registrar  to  determine  if  nonconformities  have 
been  addressed 

•  Certificate  of  conformance  to  QS-9000 

Postassessment  Phase 

•  Surveillance  audits  every  six  months 

Auditor  Criteria 

•  Third  party  registrars 

•  Successful  completion  of  QS-9000  and  quality  system  assessment  courses 

•  Registrars  accredited  for  QS-9000  compliance  by  the  RAB 

Results  Criteria 

1.  Management  responsibility 

1.1  Quality  policy 

1.2  Organization 

1.3  Management  review 

1.4  Business  plan 

1 .5  Analysis  and  use  of  company-level  data 

1.6  Customer  satisfaction 

2.  Quality  system 

2.1  General 

2.2  Quality  system  procedures 

2.3  Quality  platming 

3.  Control  review 

3.1  General 

3.2  Review 

3 .3  Amendment  to  a  contract 

3.4  Records 

4.  Design  control 

4.1  General 

4.2  Design  and  development  training 

4.3  Organizational  and  technical  interfaces 

4.4  Design  input 

4.5  Design  output 

4.6  Design  review 
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4.7  Design  verification 

4.8  Design  validation 

4.9  Design  changes 

5.  Document  and  data  control 

5.1  General 

5.2  Document  and  data  approval  and  issue 

5.3  Document  and  data  changes 

6.  Purchasing 

6.1  General 

6.2  Evaluation  of  subcontractors 

6.3  Purchasing  data 

6.4  Verification  of  purchased  product 

7.  Control  of  customer-supplied  product 

8.  Product  identification  and  traceability 

9.  Process  control 

9.1  Process  monitoring  and  operator  instructions 

9.2  Preliminary  process  capability  requirements 

9.3  Ongoing  process  performance  requirements 

9.4  Modified  preliminary  or  ongoing  capability  requirements 

9.5  Verification  of  job  set-ups 

9.6  Process  changes 

9.7  Appearance  items 

10.  Inspection  and  testing 

10.1  General 

10.2  Receiving  inspection  and  testing 

10.3  In-process  inspection  and  testing 

10.4  Final  inspection  and  testing 

10.5  Inspection  and  test  records 

1 1 .  Control  of  inspection,  measuring  and  test  equipment 

11.1  General 

1 1 .2  Control  procedure 

11.3  Inspection,  measuring  and  test  equipment  records 

1 1 .4  Measurement  system  analysis 

12.  Inspection  and  test  status 

13.  Control  of  nonconforming  product 

13.1  General 

13.2  Review  and  disposition  of  nonconforming  product 

13.3  Control  of  reworked  product 

13.4  Engineering  approved  product  authorization 
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14.  Corrective  and  preventive  action 

14.1  General 

14.2  Corrective  action 

14.3  Preventive  action 

15.  Handling,  storage,  packaging,  preservation  and  delivery 

15.1  General 

15.2  Handling 

15.3  Storage 

15.4  Packaging 

15.5  Preservation 

15.6  Delivery 

16.  Control  of  quality  records 

17.  Internal  quality  records 

18.  Training 

19.  Servicing 

20.  Statistical  techniques 

Additional  Criteria 

Sector-Specific  Requirements 

1.  Production  part  approval  process.  Suppliers  to  the  Big  3  are  required  to  prove  their 
compliance,  and  that  of  their  subcontractors,  with  the  “Production  Part  Approval 
Process”  manual. 

1.1  General 

1 .2  Engineering  change  validation 

2.  Continuous  improvement.  Suppliers  should  establish  an  approach  to  continuous 
improvement  whereby  they  can  identify  opportunities  for  improving  quality  and 
productivity  and  implement  a  methodology  to  foster  continuous  improvement. 

2.1  General 

2.2  Quality  and  productivity  improvements 

2.3  Techniques  for  continuous  improvement 

3.  Manufacturing  capabilities.  Suppliers  should  utilize  teams  to  create  the  plans 
necessary  to  meet  the  standards  of  advanced  quality  planning.  In  particular,  there 
should  be  a  methodology  in  place  to  eliminate  product  nonconformities  attributable 
to  a  design  or  process  element,  and  to  assist  in  the  design,  manufacture,  and 
inspection  of  the  tools  and  gages  required  by  the  tooling  system. 

3.1  Facilities,  equipment,  and  process  plaiming  and  effectiveness 

3.2  Mistake  proofing 
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3.3  Tool  design  and  fabrication 

3.4  Tooling  management 


Customer-Specific  Requirements 

1.  Chrysler — specific  requirements 

1.1  Third  party  registration  required.  By  31  July  1997,  all  of  Chrysler’s  suppliers 
are  to  be  third  party  QS-9()00  verified. 

1.2  Parts  identified  with  symbols.  Supplier  parts  are  to  be  marked  with  specific 
symbols  when  they  contain  and/or  represent  safety  characteristics,  special 
characteristics,  and  critical  tooling  requirements. 

1.3  Significant  characteristics.  Suppliers  may  also  identify  and  label  additional 
characteristics  that  they  deem  relevant  to  the  production  of  a  quality  product 
or  service. 

1.4  Annual  layout.  Layout  inspections  must  be  performed  for  suppliers  on  an 
annual  basis  in  order  to  confirm  that  a  supplier  remains  in  compliance  with 
Chrysler  requirements. 

1.5  Internal  quality  audits.  Suppliers  must  conduct  annual  internal  audits  of  their 
quality. 

1.6  Design  validation/production  verification.  Suppliers’  products  must  be 
verified/validated  with  regard  to  their  design  and  production  every  model 
year. 

1.7  Corrective  action  plan.  Suppliers  are  to  document  on  a  specific  Chrysler 
form  all  nonconformities  and  any  resulting  corrective  actions  made. 

1.8  Packaging,  shipping  and  labeling.  Chrysler  maintains  in  its  manuals  on 
packaging,  shipping,  and  labeling,  a  specific  set  of  instructions  that  is  to  be 
adhered  to  by  suppliers. 

1.9  Process  sign-off.  An  advance  quality  planning  team  is  required  to  perform  a 
formal  sign-off  to  indicate  a  supplier’s  process  readiness  to  manufacture  a 
new  product. 

1.10  Lot  acceptance  sampling  table.  Chrysler  provides  suppliers  with  a  table  to  be 
used  to  determine  lot  acceptance,  given  specific  characteristic  classifications. 

2.  Ford — specific  requirements 

2.1  Third  party  registration  not  required.  Ford  suppliers  are  currently  not 
required  to  obtain  third  party  verification  to  QS-9000. 

2.2  Control  item  parts.  When  characteristics  related  to  safety  and  conformance 
with  government  regulations  are  identified,  suppliers  use  standard  symbols  to 
record  them  on  drawings  and  document  them  and  any  revisions  in  the  Control 
Plan.  Ford  guidelines  also  regulate  the  use  of  the  control  item  part’s  symbol 
with  regard  to  packaging. 
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2.3  Critical  characteristics.  Characteristics  that  have  been  determined  to 
influence  the  safety  or  government  compliance  of  a  product  or  service  are 
included  on  the  Control  Plan,  along  with  any  special  assembly,  shipping,  and 
monitoring  requirements. 

2.4  Set-up  verification.  The  set-up  of  critical  characteristics  are  to  be  ensured 
through  statistical  methods. 

2.5  Control  item  fasteners.  With  regard  to  fasteners,  suppliers  must  possess  a 
system  of  control  in  analyzing  metals  for  their  hardness  and  composition  and 
ensuring  that  fastener  lots  are  traceable. 

2.6  Heat  treating.  Ford  guidelines  are  used  to  evaluate  suppliers’  characteristics 
for  heat  treating. 

2.7  Process  changes  and  design  changes  for  supplier-responsible  designs.  If  a 
supplier  would  like  to  make  a  modification  on  a  Control  Item  Part,  Ford  must 
first  review  change  information  and  grant  its  approval. 

2.8  Supplier  modification  of  control  items  and  requirements.  Control  Plan 
revisions  require  previous  Ford  approval. 

2.9  Engineering  specification  test  performance  requirements.  Suppliers  are 
required  to  use  engineering  specification  testing  to  test  if  their  design  goals 
are  being  attained.  If  not,  product  shipments  are  to  be  halted  and  customers 
notified,  until  the  appropriate  corrective  action  has  taken  place  and  been 
verified. 

2.10  System  design  specification.  Suppliers  should  evaluate  customer  needs  to 
identify  those  characteristics  for  measure  within  their  systems. 

2.11  Ongoing  process  monitoring.  Ford  details  how  a  supplier  should  determine 
the  disposition  of  a  product  when  it  has  been  produced  while  being  monitored 
through  statistical  process  control. 

2.12  Prototype  part  quality  initiatives.  If  prototypes  are  being  produced,  the 
supplier  must  have  an  established  methodology  for  evaluating  these 
prototypes  and  how  their  production  experience  may  be  used  in  planning  the 
production  process. 

2.13  QOS.  Suppliers  are  required  to  use  Ford’s  Quality  Operating  System  (QOS) 
methodology  in  the  use  of  certain  tools  and  practices,  in  order  to  extract 
increasing  levels  of  customer  satisfaction. 

2.14  Qualification  and  acceptance  criteria  for  materials.  To  qualify  materials 
initially,  suppliers  will  follow  the  appropriate  material  specification 
requirements.  Subsequent  material  qualifications  are  to  correspond  to  the 
methods  documented  and  approved  in  the  supplier’s  Control  Plan. 
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3.  General  Motors — specific  requirements 

3.1  Third  party  registration  required.  By  31  December  1997,  General  Motors’ 
suppliers  are  to  be  QS-9000  registered. 

3.2  General  procedures  and  other  requirements.  General  Motors’  suppliers  are 
required  to  adhere  to  a  number  of  additional  requirements  in  General  Motors’ 
publications  that  include:  “Key  Characteristic  Designation  System;’’ 
“Supplier  Submission  of  Material  for  Process  Approval;’’  “Component 
Verification  &  Traceability  Procedure;’’  etc. 

3.3  QS-9000  applies  to  all  General  Motors  suppliers.  All  suppliers  to  General 
Motors  are  subject  to  the  criteria  requirements  of  QS-9000. 

3.4  Customer  approval  of  control  plans.  General  Motors’  European  suppliers  are 
waived  from  having  to  obtain  approval  of  their  Control  and  Reaction  Plans. 

3.5  UPC  labeling  for  commercial  service  applications.  General  Motors 
sometimes  requires  that  suppliers  use  UPC,  rather  than  AIAG,  labeling. 

3.6  Layout  inspection  and  functional  test.  General  Motor  establishes  no  set 
frequency  at  which  the  customer  is  to  inspect  the  supplier. 


ARMY-CONTRACTOR  PERFORMANCE  CERTIFICATION  PROGRAM 

Process  Criteria 

Preliminary  Phase 

•  MSC  provides  an  introductory  (CP)2  briefing  to  the  contractor  who  has 
expressed  interest 

•  Contractor’s  senior  representative  formally  submits  a  letter  of  intent  to  enter 
the  (CP)2 

Preassessment  Phase 

•  Candidate  contractor  provides  information  on  its  government  contracts, 
facilities  and  organizational  structure 

•  All  MSCs,  with  which  the  contractor  does  and  has  done  business,  invited  to 
participate  in  the  assessment  process 

•  Contractor  and  MSC  determine  the  type  of  certification  sought  (i.e., 
production,  design/development  or  production  and  design/development) 

•  MSCs  confer  to  define  the  entities  that  are  to  be  certified 
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•  Past  performance  data  collected  from  the  contractor’s  customers 

•  Contractor  uses  (CP)2  “results  criteria”  to  perform  a  self-assessment 

Assessment  Phase 

•  Baseline  assessment  and  resulting  report  produced  by  the  formal  assessment 
team 

•  Contractor  requests  additional  assessments  as  confirmation  that  suggested 
corrective  actions  have  been  implemented 

•  All  MSC  concerns  addressed  prior  to  certification 

•  At  least  a  baseline  and  final  assessment  must  take  place 

•  Memorandum  agreement  between  government  and  contractor  detailing  the 
scope,  responsibilities  and  commitments  of  the  contractor’s  certification 

•  Zero  to  ten  scoring  system  with  which  to  assess  the  contractor  against  the  28 
certification  criteria 

•  Preliminary  through  the  completion  of  the  assessment  phase  can  take  over  2 
years 

Postassessment  Phase 

•  After  3  years  of  certification,  a  determination  is  made  as  to  whether  a  full  or 
partial  assessment  is  needed  for  certification  extension 

•  Contractor  provides  annual  review  reports  on  management  and  the  program 

•  MSC  follows  up  on  any  customer  complaints 

•  MSC  contacts  contractor  if  any  quality  problems  exist 

•  MSC  may  require  a  full  assessment  at  any  time  if  there  have  been  major 
changes,  a  decline  in  quality  improvements,  etc. 

•  Decertification  possible  for  unethical/illegal  activities  or  the  failure  to 
implement  required  corrective  actions 

Auditor  Criteria 

•  Behavior — ^high  ethical  standards;  objectivity;  sufficient  knowledge  for 
credibility;  and  flexibility  in  dealing  with  busy  demands  of  those  being 
reviewed 

•  Teamwork — ^work  effectively  with  the  personnel  of  the  contractor  being 
assessed;  and  promote  the  concept  of  working  toward  the  same  objective 
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•  Communications — skillful  at  presenting  ideas  and  recommendations;  ability 
to  keep  an  open  mind;  strong  listening  skills;  and  avoid  making  judgments 
before  all  information  is  understood 

•  Persuasion — present  persueisive  case  for  the  contractor’s  need  to  make 
corrective  actions 

•  Qualifications — ^formal  education  in  assessment  techniques  and  quality 
standards;  and  possess  ASQC  certification  or  RAB  registration  accreditation. 

Results  Criteria 

1.  Management  responsibility 

2.  Quality  system 

3.  Contract  review 

4.  Design  control 

5.  Document  and  data  control 

6.  Purchasing 

7.  Control  of  customer — supplied  product 

8.  Product  identification  and  traceability 

9.  Process  control 

10.  Inspection  and  testing 

11.  Control  of  inspection,  measuring,  and  test  equipment 

12.  Inspection  and  test  status 

13.  Control  of  nonconforming  product 

14.  Corrective  and  preventive  action 

15.  Handling,  storage,  packaging,  preservation  and  delivery 

16.  Control  of  quality  records 

17.  Internal  quality  audits 

18.  Training 

19.  Servicing 

20.  Statistical  techniques 

21.  Customer  satisfaction.  Contractors’  employees  at  all  levels  must  be  aware  of  who 
the  customers  are  and  there  should  be  a  formal  system  whereby  customers  may 
communicate  with  the  contractor  and/or  voice  any  concerns. 

22.  Quality  costs.  Contractors  are  to  maintain  documentation  on  the  costs  they  incur  due 
to  their  quality  system  and  its  relationship  to  total  costs. 
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23.  Warranty  performance.  A  system  is  required  for  processing  warranties  and  ensuring 
that  appropriate  corrective  measures  are  being  made  by  the  contractor. 

24.  Ethics.  A  contractor  must  have  an  established  ethical  standard  of  conduct,  which  is 
openly  communicated  to  employees  and  customers  and  contains  specific  reference  to 
any  government  business  dealings. 

25.  Business  planning.  The  contractor  is  to  use  short-  and  long-term  business  planning 
with  an  ongoing  focus  on  improvements  in  order  to  design,  implement,  and  evaluate 
its  business  strategy. 

26.  Safety.  A  contractor  is  required  to  possess  a  safety  process  that:  1)  is  conununicated 
to  all  employees;  2)  includes  the  necessary  protective  equipment  for  employees;  3) 
provides  employees  with  a  mechanism  to  report  safety  violations;  and  4)  adheres  to 
all  federal,  state,  and  local  safety  regulations. 

27.  Environmental.  Contractors  need  to  demonstrate  their  possession  of  a  process  for 
environmental  compliance,  which  outlines  the  process  for  controlling  hazardous 
materials,  allows  employees  to  report  environmental  violations,  and  adheres  to  all 
federal,  state,  and  local  environmental  regulations. 

28.  Continuous  improvement  program.  A  contractor  is  required  to  have  a  documented 
policy  for  a  continuous  improvement  process,  which  is  administered  at  the  senior 
management  level.  This  process  includes  the  setting  of  short-  and  long-term  goals 
and  the  suggestion  of  specific  metrics  to  be  used  to  measure  trends. 

FAA-AIRCRAFT  CERTIFICATION  SYSTEMS  EVALUATION  PROGRAM 

Process  Criteria 

•  Evaluation  conducted  by  evaluator  team 

•  Team  leader  prepares  report  and  forwards  to  the  manufacturing  inspection 
office 

•  Report  then  given  to  the  principal  inspector  in  charge  of  the  facility 

•  Post-evaluation  conference  held  with  the  facility’s  management  and  issues, 
results,  and  findings  are  reviewed 

•  Principal  inspector  responsible  for  surveillance  of  facility  in  the  event  of 
formal  corrective  actions 

•  Evaluation  data  entered  into  the  ACSEP  data  base 

Auditor  Criteria 

•  Auditors  selected  on  the  basis  of  educational  and  experiential  criteria 

•  Evaluator  training  through  FAA  Academy  and  Aircraft  Certification  Service 
(AIR)  instructors  who  are  experienced  evaluation  team  leaders 
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•  Mandatory  participation  in  a  minimum  number  of  ACSEP  evaluations  as  a 
trainee 

•  Teams  consist  of  personnel  from  FAA  engineering,  flight  test,  and 
manufacturing  inspection 

Results  Criteria 

Major  Systems 

1.  Quality 

2.  Engineering 

3.  Communications  with  the  FAA 

4.  Manufacturing 

5.  Service  and  product 

6.  Management 

Subsystems 

1.  Organization  and  responsibility 

2.  Design  data  control 

3.  Software  quality  assurance 

4.  Manufacturing  processes 

5.  Special  manufacturing  processes 

6.  Statistical  quality  control 

7.  Nonconforming  material 

8.  Material  handling  and  storage 

9.  Nondestructive  inspection 

10.  Tool  and  gauge 

11.  Testing 

12.  Internal  audit 

13.  Global  production 

14.  Supplier  control 

15.  Airworthiness  determination 

16.  FAR  reporting  requirements 

17.  Manufacturing  maintenance  facility 
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DSCC— QUALIFIED  MANUFACTURER’S  LIST 
Process  Criteria 

•  Specification  and  its  qualification  requirements  presented  to  industry 

•  Manufacturer  requests  qualification  for  the  specification 

•  Manufacturing  facilities  audit  is  performed  by  government  employees 

•  Qualification  examination  and  testing  specific  to  the  specification 

•  Manufacturer  notified  as  to  the  results  of  testing  (if  qualification  testing  was 
passed,  then  the  letter  details  the  processes  and  materials  to  be  placed  on  the 
specification’s  QML,  otherwise  the  letter  provides  a  detailed  explanation  as  to 
the  qualification  testing  failed) 

•  Letter  of  notification 

•  At  time  of  2-year  review  of  qualification,  the  manufacturer  must:  1)  certify  its 
qualification;  2)  submit  to  periodic  re-testing;  or  3)  undertake  a  complete 
requalification  process 

Auditor  Criteria 
Not  specified 

Results  Criteria 

Criteria  vary  according  to  a  specification’s  qualification  requirements. 

DISC— QSLD/QSLM 
Process  Criteria 
Preassessment  Phase 

•  Supplier  requests  application  form  and  criteria/provisions  fi'om  DISC 

•  Supplier  must  possess  a  quality  program  that  meets  program  criteria,  a  single 
quality  control  program  and  a  CAGE  code 

•  Supplier  submits  application  including  its  quality  manual 
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Assessment  Phase 

•  DISC  evaluates  application  against  specific  program  criteria 

•  DISC  assesses  any  of  the  supplier’s  recent  government  and  industry 
surveys/audits  to  determine  if  a  facility  site  visit  is  necessary 

•  DISC  performs  site  surveys  to  review  the  quality  control  program  and  systems 
in  place  with  regard  to  criteria 

•  If  noncompliance  is  discovered,  DISC  issues  a  Letter-Note  of  E>enial  of 
Qualification,  which  cites  the  reasons  for  denial  and  may  outline  corrective 
actions  to  be  taken  for  qualification  approval 

•  Once  compliance  is  determined,  DISC  issues  a  Letter-Notice  of  Qualification 
to  the  supplier 

Postassessment  Phase 

•  DISC  conducts  periodic  announced  and  unannounced  post-award  audits  to 
reaffirm  compliance  with  program  criteria 

•  Post-award  audits  include  the  independent  lab  testing  of  a  random  sampling  of 
the  supplier’s  product 

•  Qualification  lapses  in  3  years  with  a  requalification  application  required  120 
days  prior  to  expiration 

Auditor  Criteria 

•  Site  surveys  and  audits  conducted  by  DISC  agents 

Results  Criteria 

The  specific  results  criteria  vary  depending  upon  the  QSLD/QSLM  program  in  question. 

The  following  are  several  examples  of  these  types  of  criteria  for  the  various  QSLD/QSLD 

programs: 

Bulk  Metals  QSLD 

3.1  Management  responsibility 

3.2  Document  control 

3.3  Purchasing 

3.4  Product  traceability 

3.5  Process  control 
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3.6  Inspection  of  material  receipts 

3.7  Test  control 

3.8  Test  and  measurement  equipment 

3.9.1  Non-conforming  material 

3.9.2  Corrective  action 

3.10  Storage,  packaging  and  shipping 

3.11  Records  control 

3.12  Audits 

3.13  Personnel  training 

Class  3  Threaded  Fasteners  QSLD 

3 . 1  Management  responsibility 

3.2  Document  control 

3.3  Purchasing 

3.4  Product  traceability 

3.5  Lot  control  and  marking 

3.6  Inspection  of  material 

3.7  Test  control 

3.8  Test  and  measinement  equipment 

3.9.1  Non-conforming  material 

3.9.2  Corrective  action 

3.10  Storage,  packaging,  and  shipping 

3.11  Records  control 

3.12  Audits 

3.13  Personnel  training 

3.14  Products 

Class  3  Threaded  Fasteners  QSLM 

3.1  Management  responsibility 

3.2  Document  control 

3.3  Purchasing 

3.4  Product  traceability 

3.5  Lot  control  and  marking 
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3.6  Process  control 

3.7  Inspection  of  material 

3.8  Test  control 

3.9  Test  and  measurement  equipment 

3 . 1 0. 1  Non-conforming  action 

3.10.2  Corrective  action 

3.11  Storage,  packaging,  and  shipping 

3.12  Records  control 

3.13  Audits 

3.14  Personnel  training 

3.15  Products 
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Appendix  B 

QUALITY  AWARD  CRITERIA 


MALCOLM  BALDRIGE  NATIONAL  QUALITY  AWARD 

1.  Process  Criteria 

•  Application  independently  reviewed  by  six  examiners  and  panel  of  judges 
determines  who  goes  forward 

•  Application  jointly  reviewed  by  at  least  six  examiners  and  panel  of  judges 
determines  who  goes  forward 

•  Site  visits  by  at  least  five  examiners  and  one  senior  examiner  and  site  report 
submitted  to  the  panel  of  judges 

•  Final  review  and  recommendations  made  by  the  panel  of  judges 

•  Award  recipients  determination  made  by  the  Secretary  of  Commerce 

•  Three-dimensional  scoring  system  (approach,  deployment,  and  results) 

•  Five-level  scoring  guidelines 

•  Feedback  report  for  every  nominee 

2.  Auditor  Criteria 

•  Must  apply  annually 

•  Breadth  of  experience 

•  Diversity  of  experience 

•  Leadership  and  external  representation 

•  Knowledge  of  business,  specialized  areas  or  quality  practices  and 
improvement  strategies 

•  Two  references 

•  3-day  preparation  course 

•  20-  to  40-hour  case  study  evaluation 
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•  50  senior  examiners  selected  from  annual  examiner  pool  by  Secretary  of 
Commerce 

•  Nine  members  of  the  panel  of  judges  selected  from  annual  examiner  pool  by 
the  Secretary  of  Commerce 

3.  Results  Criteria 

The  numbers  in  parentheses  after  the  category  are  the  number  of  points  allocated 
to  that  category. 

•  Leadership  (110) 

—  Leadership  system  (80) 

—  Company  responsibility  and  citizenship  (30) 

•  Strategic  planning  (80) 

—  Strategy  development  process  (40) 

—  Company  strategy  (40) 

•  Customer  and  market  focus  (80) 

—  Customer  and  market  knowledge  (40) 

—  Customer  satisfaction  and  relationship  enhancement  (40) 

•  Information  and  analysis  (80) 

—  Selection  and  use  of  information  and  data  (25) 

—  Selection  and  use  of  comparative  information  and  data  (15) 

—  Analysis  and  review  of  company  performance  (40) 

•  Human  resource  development  and  management  (100) 

—  Work  systems  (40) 

—  Employee  education,  training  and  development  (30) 

—  Employee  well-being  and  satisfaction  (30) 

•  Process  management  (100) 

—  Management  of  product  and  service  processes  (60) 

—  Management  of  support  processes  (20) 

—  Management  of  supplier  and  partnering  processes  (20) 
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•  Business  results  (450) 

—  Customer  satisfaction  results  ( 1 30) 

—  Financial  and  market  results  ( 1 30) 

—  Human  resource  results  (35) 

—  Supplier  and  partner  results  (25) 

—  Company  specific  results  ( 1 30) 

DEMING  APPLICATION  PRIZE  OF  THE  JAPANESE  UNION  OF  JAPANESE 
SCIENTISTS  AND  ENGINEERS 

1.  Process  Criteria 

•  Applicants  self-nominate  by  addressing  the  award  criteria  and  providing  an 
additional  status  report 

•  Application  reviewed  by  the  application  prize  subcommittee  with  results 
forwarded  to  the  chair  of  the  Deming  prize  conmiittee 

•  Applicants  may  deliver  a  one-hour  briefing  to  the  evaluating  subcommittee 

•  At  least  two  examiners  conduct  “on-the-spot”  survey  of  those  nominees 
designated  by  the  committee 

•  Application  prize  subcommittee  presents  a  report  of  recommendations 

•  Deming  prize  committee  makes  final  award  decisions 

2.  Auditor  Criteria 

•  Deming  prize  committee  members,  who  are  from  industry,  media,  and  the 
government,  are  unpaid  appointees  by  the  chair  (president  of  the  JUSE) 

3.  Results  Criteria 

•  Company’s  policy 

•  Organization  and  its  management 

•  Education  and  dissemination 

•  Implemental  conditions 

—  The  collection,  transmittal  and  eventual  utilization  of  quality  information 
—  Analysis 
—  Standardization 
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—  Control 
—  Effect 
—  Future  planning 

PRESIDENT’S  AWARD  FOR  QUALITY  AND  PRODUCTIVITY 
IMPROVEMENT 

1.  Process  Criteria 

•  Application  reviewed  by  government  agency  before  submission  to  0PM 

•  Preliminary  review  by  OPM  to  ensure  applicant’s  eligibility 

•  Applications  reviewed  jointly  by  examiner  teams  and  selections  made  for  site 
visits 

•  Site  visit  review  by  an  examiner  team  and  report  submitted  to  the  panel  of 
judges 

•  Panel  of  judges  conducts  the  final  review  and  makes  recommendations  to  the 
OPM 

•  The  president  makes  final  decision  on  award  recipients 

•  Three-dimensional  scoring  system  (approach,  deployment,  and  results) 

•  Five-level  scoring  guidelines 

•  Feedback  report  for  every  nominee 

2.  Auditor  Criteria 

•  Examiners  represent  public  and  private  sector  organizations 

3.  Results  Criteria 

The  number  in  parentheses  after  the  category  is  the  number  of  points  allocated  to 
that  category. 

•  Leadership  (120) 

—  Senior  executive  leadership  (60) 

—  Leadership  system  and  organization  (30) 

—  Public  responsibility  and  corporation  citizenship  (30) 
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•  Information  and  analysis  (75) 

—  Management  of  information  and  data  (20) 

—  Competitive  comparisons  and  benchmarking  (15) 

—  Analysis  and  use  of  organization-level  data  (40) 

•  Strategic  planning  (55) 

—  Strategy  development  (35) 

—  Strategy  deployment  (20) 

•  Human  resource  development  and  management  ( 140) 

—  Human  resource  planning  and  evaluation  (20) 

—  High  performance  work  systems  (45) 

—  Employee  education,  training,  and  development  (50) 

—  Employee  well-being  and  satisfaction  (25) 

•  Process  management  (140) 

—  Design  and  introduction  of  products  and  services  (40) 

—  Process  management:  product  and  service  production  and  delivery  (40) 
—  Process  management:  support  services  (30) 

—  Management  of  supplier  performance  (30) 

•  Business  results  (220) 

—  Product  and  service  quality  results  (65) 

—  Organization  operational  and  financial  results 
—  Human  resource  results  (30) 

—  Supplier  performance  results  (25) 

•  Customer  focus  and  satisfaction  (250) 

—  Customer  knowledge  (30) 

—  Customer  relationship  management  (30) 

—  Customer  satisfaction  determination  (30) 

—  Customer  satisfaction  results  (160) 
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GEORGE  M.  LOW  AWARD:  NASA’S  QUALITY  AND  EXCELLENCE  AWARD 


1.  Process  Criteria 

•  Nominations  are  presented  by  the  NASA  centers 

•  NASA  strategic  enterprise  GML  review  council  screens  nominees  and 
forwards  10  finalists 

•  GML  validation  board  evaluates  supplemental  responses,  performs  site  visits, 
and  recommends  recipients 

•  GML  panel  of  judges  selects  winner(s) 

•  Winner(s)  are  approved  by  the  NASA  administrator 

2.  Auditor  Criteria 

•  Strategic  enterprise  review  council:  4  members  representing  the  strategic 
enterprises  (determined  through  evaluation  process  by  the  enterprise  quality 
council  and  endorsed  by  their  enterprise  associate  administrators) 

•  Validation  board:  5-member  board  composed  of  3  strategic  enterprise 
representatives  and  2  representatives  from  the  headquarters  office 

•  Panel  of  judges:  3  judges — 2  members  from  the  strategic  enterprises  and  1 
associate  administrator 

3.  Results  Criteria 

The  number  in  parentheses  after  the  category  is  the  number  of  points  allocated  to 
that  category. 

•  NASA  contract  performance  and  customer  satisfaction  (300) 

—  Contract  technical  performance  and  outcomes  (200) 

—  Customer  knowledge,  relationships,  and  value  (100) 

•  NASA  schedule  (150) 

•  NASA  cost  (150) 

•  Long-term  organizational  initiatives  to  respond  to  NASA’s  strategic 
aspirations  (100) 

—  Strategic  planning  (50) 

—  Long-term  research  and  development  (50) 
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•  Leadership  and  continuous  improvement  (100) 

•  Innovative  management  and  technology  breakthroughs  (100) 

•  Items  of  special  interest  to  NASA  (100) 

SmGEO  SfflNGO  PRIZE  FOR  EXCELLENCE  IN  MANUFACTURING 

1.  Process  Criteria 

•  Nominee  “achievement  reports”  evaluated  individually  and  jointly  by  board 
of  examiner  members 

•  Site  verifications  conducted  by  teams  of  5  or  6  examiners 

•  Board  of  examiners  recommends  recipients  to  Shingo  Prize  council 

•  Shingo  Prize  council  ratifies  recommendations 

•  Feedback  reports  for  every  nominee 

•  Two-dimensional  evaluation  (strategy/implementation  and  results) 

•  Four-level  scoring  guidelines 

2.  Auditor  Criteria 

•  Board  of  Examiners  are  practitioners  and  academicians  with  knowledge  of 
manufacturing  improvement  methods,  systems,  processes  and  research 

•  Examiners  represent  leading  business,  government  and  professional  agencies, 
and  academics 

•  Shingo  Prize  provides  training  for  examiner  preparation 

3.  Results  Criteria 

The  number  in  parentheses  after  the  category  is  the  number  of  points  allocated  to 
that  category. 

•  Total  quality  and  productivity  management  culture  and  infrastructure  (275) 
—  Leading  (100) 

—  Empowering  (100) 

—  Partnering  (75) 
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•  Manufacturing  strategy,  processes,  and  systems  (425) 
—  Manufacturing  vision  and  strategy  (50) 

—  Manufacturing  process  integration  (125) 

—  Quality  and  productivity  methods  integration  (125) 
—  Manufacturing  and  business  integration  (125) 

•  Measured  quality  and  productivity  (200) 

—  Quality  enhancement  (100) 

—  Productivity  improvement  ( 1 00) 

•  Measured  customer  service  (100) 

—  Customer  satisfaction  ( 1 00) 
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Appendix  C 

ISSUES  AND  CONSIDERATIONS 

When  considering  recognition  or  reciprocal  concepts  in  lieu  of  a  full  customer 
oversight  effort,  buyers  have  to  establish  their  confidence  in  the  value  of  another  buyer’s 
oversight.  In  doing  this,  some  very  important  factors  must  be  considered. 

1.  Certification  Process  Management 

•  Does  the  certifier  use  audit  plans  and,  if  so,  how  thorough  is  the  audit  plan? 

•  Do  the  audit  reports  indicate  that  the  audit  plan  was  followed?  Are  the  audit 
reports  well  written  and  do  they  contain  valuable  information? 

•  What  is  the  frequency  of  internal  audits? 

•  How  timely  are  audits  closed  out? 

•  What  is  the  follow-up  procedure  and  how  effective  is  the  corrective  action? 

•  How  many  corrective  actions  were  issued  during  the  certification  process, 
and  how  long  did  the  contractor  take  to  make  the  corrections? 

•  Are  the  audit  records  traceable  and  recoverable? 

•  How  effectively  are  metrics  used  to  measure  the  potency  of  internal  audits? 

•  What  is  required  for  the  contractor  to  maintain  certification? 

•  How  often  is  a  contractor  reaudited  for  recertification? 

•  How  extensive  is  the  reaudit  process?  Full  scope  or  partial? 

•  Is  there  a  process  in  place  to  revoke  certification  of  suppliers  or  contractors 
who  fail  to  maintain  the  standards  of  certification  above  ISO  9000?  If  so, 
how  many  organizations  have  lost  their  certification  and  under  what 
conditions? 

2.  Auditor  Qualification 

•  What  are  the  minimum  qualifications  required  to  become  an  auditor? 
Consider:  education;  training;  work  experience;  quality  experience;  auditing 
experience. 
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•  Are  the  auditors  ISO-certified,  or  just  qualified?  Are  they  certified  to  another 
program’s  criteria,  such  as  NADCAP? 

•  Have  the  auditors  passed  a  RAB  approved  exam? 

•  Does  the  certifying  organization  conduct  personal  interviews  with  the 
auditors  prior  to  accepting  them? 

•  What  are  the  requirements  for  auditors  to  maintain  their  competence  and 
thereby  maintain  their  status? 

•  Under  what  conditions  would  an  auditor  lose  his  or  her  certification? 

3.  Certification  Results  Criteria 

•  Does  the  program  evaluate  management  effectively? 

—  Communication  of  quality  policy  to  all  levels 
—  Existence  of  policy  and  guidance  for  all  quality  improvement  efforts 
—  Allocation  of  resources  to  support  continuous  improvement 
—  Management  review  of  quality  program 
—  Teaming  activities  reporting  results  to  management 

•  Does  the  certifier  adequately  assess  the  contractor’s  risk  management  ability? 

•  What  importance  is  given  to  the  determination  and  use  of  critical 
characteristics? 

•  How  well  does  the  certifier  assess  the  contractor’s  ability  to  assure  that  all 
levels  of  the  organization  are  aware  of  who  their  customers  are— internal  and 
external? 

•  Are  the  continuous  improvement  process  criteria  evaluated? 

•  What  metrics  does  the  certifier  employ  to  assess  the  quality  management 
system?  Are  they  sufficient? 

•  How  extensively  does  the  certifier  use  other  metrics,  such  as  the 
accomplishment  of  predicted  schedule/costs/operations  and  support  costs,  or 
the  success  rate  in  solving  the  problems  with  vital  parts/subsystems? 

•  What  methodology  is  used  by  the  certifier  to  evaluate  the  existence  and 
effectiveness  of  supplier  empowerment  and  the  customer/supplier 
partnership? 
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DEFINITIONS  FROM  ISO  GUIDE,  STANDARDIZATION  AND 
RELATED  ACTIVITIES-GENERAL  VOCABULARY 


Conformity  assessment.  Any  activity  concerned  with  determining  directly  or  indirectly 
that  relevant  requirements  are  fulfilled.* 

Registration.  Procedure  by  which  a  body  indicates  relevant  characteristics  of  a  product, 
process  or  service,  or  particulars  of  a  body  or  person,  in  an  appropriate,  publicly 
available,  list. 

Accreditation.  Procedure  by  which  an  authoritative  body  gives  formal  recognition  that  a 
body  or  person  is  competent  to  carry  out  specific  tasks. 

Reciprocity.  Bilateral  relationship  where  both  parties  have  the  same  rights  and 
obligations  towards  each  other 

Certification.  Procedure  by  which  a  third  party  gives  written  assurance  that  a  product, 
process,  or  service  conforms  to  specified  requirements. 

Recognition  arrangement.  Agreement  that  is  based  on  the  acceptance  by  one  party  of 
results,  presented  by  another  party,  from  the  implementation  of  one  or  more  designated 
functional  elements  of  a  conformity  assessment  system.^ 


*  Typical  examples  of  conformity  assessment  activities  are  sampling,  testing  and  inspection;  evaluation, 
verification,  and  assurance  of  conformity  (supplier’s  declaration,  certification);  registration, 
accreditation,  and  approval  as  well  as  their  combinations. 

^  Reciprocity  can  exist  within  a  multilateral  arrangement  comprising  a  network  of  bilateral  reciprocal 
relationships. 

Though  rights  and  obligations  are  the  same,  opportunities  emanating  from  them  may  differ;  this  may 
lead  to  unequal  relations  between  parties. 

^  Typical  examples  of  recognition  arrangements  are  testing  arrangements,  inspection  arrangements,  and 
certification  arrangements. 

Recognition  arrangements  may  be  established  at  national,  regional,  or  international  level. 

An  agreement  limited  to  declaration  of  equivalence  of  procedures  without  acceptance  of  results  does 
not  meet  the  above  definition. 
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ACRONYMS 


AAQG 

ACSEP 

AECMA 

AMC 

ANSI 

AQP 

AQS 

ASQ 

BQS 

CAGE 

C.A.S.E. 

CDD 

CES 

(CP)2 

CPARS 

CQCS 

DARS 

DISC 

DLA 

DMS 

DoD 

DRS 

DSCC 

EN 

FAA 

FAR 


American  Aerospace  Quality  Group 

Aircraft  Certification  Systems  Evaluation  Program 

European  Association  of  Aerospace  Industries 

Army  Materiel  Command 

American  National  Standards  Institute 

Advanced  Quality  Practices 

Advanced  Quality  System 

American  Society  for  Quality 

Basic  Quality  System 

Commercial  and  Government  Entity  Code 

Coordinating  Agency  for  Supplier  Evaluation 

Contract  Delivery  Data 

Contractor  Evaluation  System 

Contractor  Performance  Certification  Program 

Contractor  Performance  Assessment  Reporting  System 

Common  Quality  Certification  System 

Defense  Acquisition  Regulation  Supplement 

Defense  Industrial  Supply  Center 

Defense  Logistics  Agency 

Depot  Maintenance  System 

Department  of  Defense 

Deficiency  Reporting  System 

Defense  Supply  Center  Columbus 

European  Standards 

Federal  Aviation  Administration 

Federal  Acquisition  Regulation 
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FSC 

Federal  Supply  Classification 

GIQLP 

Government  and  Industry  Quality  Liaison  Panel 

GML 

George  M.  Low  Award 

GV/BB 

Greatest  Value/Best  Buy 

HMO 

Health  Maintenance  Organization 

IDA 

Institute  for  Defense  Analyses 

INFAC 

Instrumented  Factory  for  Gears 

IPDS 

Integrated  Product  Development  System 

IPT 

Integrated  Product  Team 

ISO 

International  Organization  for  Standardization 

MIR 

Material  Inspection  Records 

MOA 

Memorandum  of  Agreement 

MSC 

Major  Subordinate  Command 

MOU 

Memorandum  of  Understanding 

NADCAP 

National  Aerospace  and  Defense  Contractors  Accreditation 
Program 

NAM 

National  Association  of  Manufacturers 

NAP 

National  Accreditation  Program 

NASA 

National  Aeronautics  and  Space  Administration 
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PPI 

Past  Performance  Information 
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